Sep 18 19:38:03.368337 ip-10-0-128-161 systemd[1]: Starting Kubernetes Kubelet...
Sep 18 19:38:03.781659 ip-10-0-128-161 kubenswrapper[2564]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 19:38:03.781659 ip-10-0-128-161 kubenswrapper[2564]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 18 19:38:03.781659 ip-10-0-128-161 kubenswrapper[2564]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 19:38:03.782305 ip-10-0-128-161 kubenswrapper[2564]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 18 19:38:03.782305 ip-10-0-128-161 kubenswrapper[2564]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 19:38:03.782620 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.782515    2564 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 18 19:38:03.786357 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786340    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:38:03.786399 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786360    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:38:03.786399 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786364    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:38:03.786399 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786368    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:38:03.786399 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786371    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:38:03.786399 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786374    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:38:03.786399 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786377    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:38:03.786399 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786380    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:38:03.786399 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786384    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:38:03.786399 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786386    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:38:03.786399 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786390    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:38:03.786399 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786392    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:38:03.786399 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786396    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:38:03.786399 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786399    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:38:03.786399 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786402    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:38:03.786399 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786405    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:38:03.786399 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786408    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:38:03.786784 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786411    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:38:03.786784 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786414    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:38:03.786784 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786417    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:38:03.786784 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786420    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:38:03.786784 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786422    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:38:03.786784 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786425    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:38:03.786784 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786428    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:38:03.786784 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786431    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:38:03.786784 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786434    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:38:03.786784 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786437    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:38:03.786784 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786439    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:38:03.786784 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786442    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:38:03.786784 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786444    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:38:03.786784 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786447    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:38:03.786784 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786450    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:38:03.786784 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786452    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:38:03.786784 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786456    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:38:03.786784 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786459    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:38:03.786784 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786461    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:38:03.787254 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786464    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:38:03.787254 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786466    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:38:03.787254 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786469    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:38:03.787254 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786472    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:38:03.787254 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786475    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:38:03.787254 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786477    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:38:03.787254 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786481    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:38:03.787254 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786483    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:38:03.787254 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786486    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:38:03.787254 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786489    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:38:03.787254 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786491    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:38:03.787254 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786494    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:38:03.787254 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786497    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:38:03.787254 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786499    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:38:03.787254 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786503    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:38:03.787254 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786505    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:38:03.787254 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786508    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:38:03.787254 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786511    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:38:03.787254 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786513    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:38:03.787254 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786516    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:38:03.787749 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786519    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:38:03.787749 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786521    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:38:03.787749 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786524    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:38:03.787749 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786530    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:38:03.787749 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786534    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:38:03.787749 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786538    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:38:03.787749 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786541    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:38:03.787749 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786544    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:38:03.787749 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786547    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:38:03.787749 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786549    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:38:03.787749 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786560    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:38:03.787749 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786565    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:38:03.787749 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786569    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:38:03.787749 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786572    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:38:03.787749 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786575    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:38:03.787749 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786579    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:38:03.787749 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786582    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:38:03.787749 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786585    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:38:03.787749 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786588    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:38:03.788227 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786592    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:38:03.788227 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786595    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:38:03.788227 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786598    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:38:03.788227 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786600    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:38:03.788227 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786603    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:38:03.788227 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786606    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:38:03.788227 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786608    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:38:03.788227 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786612    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:38:03.788227 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786615    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:38:03.788227 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786619    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:38:03.788227 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786623    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:38:03.788227 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786626    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:38:03.788227 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786629    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:38:03.788227 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786631    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:38:03.788227 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786634    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:38:03.788227 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786637    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:38:03.788227 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786639    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:38:03.788227 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786642    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:38:03.788227 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786644    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:38:03.788227 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786647    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:38:03.788714 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786650    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:38:03.788714 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786652    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:38:03.788714 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786655    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:38:03.788714 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.786657    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:38:03.788714 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787394    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:38:03.788714 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787401    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:38:03.788714 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787404    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:38:03.788714 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787407    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:38:03.788714 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787412    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:38:03.788714 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787416    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:38:03.788714 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787420    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:38:03.788714 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787423    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:38:03.788714 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787425    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:38:03.788714 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787428    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:38:03.788714 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787431    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:38:03.788714 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787433    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:38:03.788714 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787436    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:38:03.788714 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787439    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:38:03.788714 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787441    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:38:03.788714 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787445    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:38:03.789211 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787448    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:38:03.789211 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787450    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:38:03.789211 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787453    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:38:03.789211 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787456    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:38:03.789211 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787458    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:38:03.789211 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787461    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:38:03.789211 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787464    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:38:03.789211 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787466    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:38:03.789211 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787469    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:38:03.789211 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787472    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:38:03.789211 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787475    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:38:03.789211 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787477    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:38:03.789211 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787480    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:38:03.789211 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787483    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:38:03.789211 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787485    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:38:03.789211 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787488    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:38:03.789211 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787490    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:38:03.789211 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787492    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:38:03.789211 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787496    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:38:03.789211 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787499    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:38:03.789697 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787501    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:38:03.789697 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787504    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:38:03.789697 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787507    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:38:03.789697 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787510    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:38:03.789697 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787513    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:38:03.789697 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787515    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:38:03.789697 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787518    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:38:03.789697 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787521    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:38:03.789697 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787524    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:38:03.789697 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787529    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:38:03.789697 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787533    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:38:03.789697 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787536    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:38:03.789697 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787539    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:38:03.789697 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787542    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:38:03.789697 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787545    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:38:03.789697 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787548    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:38:03.789697 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787551    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:38:03.789697 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787554    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:38:03.789697 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787556    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:38:03.789697 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787559    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:38:03.790200 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787562    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:38:03.790200 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787565    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:38:03.790200 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787567    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:38:03.790200 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787570    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:38:03.790200 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787572    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:38:03.790200 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787576    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:38:03.790200 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787579    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:38:03.790200 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787582    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:38:03.790200 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787585    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:38:03.790200 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787589    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:38:03.790200 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787592    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:38:03.790200 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787594    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:38:03.790200 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787597    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:38:03.790200 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787600    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:38:03.790200 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787602    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:38:03.790200 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787605    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:38:03.790200 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787609    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:38:03.790200 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787612    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:38:03.790200 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787614    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:38:03.790710 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787617    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:38:03.790710 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787620    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:38:03.790710 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787622    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:38:03.790710 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787626    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:38:03.790710 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787628    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:38:03.790710 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787632    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:38:03.790710 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787635    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:38:03.790710 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787637    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:38:03.790710 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787640    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:38:03.790710 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787643    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:38:03.790710 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787645    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:38:03.790710 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787648    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:38:03.790710 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787650    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:38:03.790710 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787653    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:38:03.790710 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787656    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:38:03.790710 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787658    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:38:03.790710 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787661    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:38:03.790710 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787663    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:38:03.790710 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787666    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:38:03.791276 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787668    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:38:03.791276 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787671    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:38:03.791276 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787673    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:38:03.791276 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787676    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:38:03.791276 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.787679    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:38:03.791276 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788078    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:38:03.791276 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788083    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:38:03.791276 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788086    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:38:03.791276 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788089    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:38:03.791276 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788093    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:38:03.791276 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788095    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:38:03.791276 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788098    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:38:03.791276 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788102    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:38:03.791276 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788104    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:38:03.791276 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788107    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:38:03.791276 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788110    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:38:03.791276 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788115    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:38:03.791276 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788118    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:38:03.791276 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788121    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:38:03.791749 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788124    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:38:03.791749 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788126    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:38:03.791749 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788129    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:38:03.791749 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788132    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:38:03.791749 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788135    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:38:03.791749 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788138    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:38:03.791749 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788140    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:38:03.791749 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788143    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:38:03.791749 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788145    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:38:03.791749 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788148    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:38:03.791749 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788151    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:38:03.791749 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788153    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:38:03.791749 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788156    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:38:03.791749 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788158    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:38:03.791749 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788161    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:38:03.791749 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788164    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:38:03.791749 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788167    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:38:03.791749 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788169    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:38:03.791749 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788172    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:38:03.791749 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788175    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:38:03.792292 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788177    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:38:03.792292 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788180    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:38:03.792292 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788183    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:38:03.792292 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788186    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:38:03.792292 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788189    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:38:03.792292 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788192    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:38:03.792292 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788195    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:38:03.792292 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788199    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:38:03.792292 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788202    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:38:03.792292 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788208    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:38:03.792292 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788211    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:38:03.792292 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788213    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:38:03.792292 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788216    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:38:03.792292 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788219    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:38:03.792292 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788221    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:38:03.792292 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788224    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:38:03.792292 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788227    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:38:03.792292 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788229    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:38:03.792292 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788232    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:38:03.792776 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788234    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:38:03.792776 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788237    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:38:03.792776 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788240    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:38:03.792776 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788242    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:38:03.792776 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788245    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:38:03.792776 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788248    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:38:03.792776 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788250    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:38:03.792776 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788253    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:38:03.792776 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788255    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:38:03.792776 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788258    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:38:03.792776 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788261    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:38:03.792776 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788265    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:38:03.792776 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788268    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:38:03.792776 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788270    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:38:03.792776 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788273    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:38:03.792776 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788275    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:38:03.792776 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788278    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:38:03.792776 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788282    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:38:03.792776 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788285    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:38:03.793287 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788288    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:38:03.793287 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788291    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:38:03.793287 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788294    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:38:03.793287 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788297    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:38:03.793287 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788301    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:38:03.793287 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788304    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:38:03.793287 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788307    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:38:03.793287 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788310    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:38:03.793287 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788312    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:38:03.793287 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788315    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:38:03.793287 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788317    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:38:03.793287 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788320    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:38:03.793287 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788322    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:38:03.793287 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788325    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:38:03.793287 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788327    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:38:03.793287 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788330    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:38:03.793287 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788332    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:38:03.793287 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788335    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:38:03.793287 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788337    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:38:03.793287 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788340    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:38:03.793812 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788342    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:38:03.793812 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788345    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:38:03.793812 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788350    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:38:03.793812 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788352    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:38:03.793812 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788355    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:38:03.793812 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788358    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:38:03.793812 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.788360    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:38:03.793812 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788438    2564 flags.go:64] FLAG: --address="0.0.0.0"
Sep 18 19:38:03.793812 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788488    2564 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 18 19:38:03.793812 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788496    2564 flags.go:64] FLAG: --anonymous-auth="true"
Sep 18 19:38:03.793812 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788501    2564 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 18 19:38:03.793812 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788507    2564 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 18 19:38:03.793812 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788510    2564 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 18 19:38:03.793812 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788514    2564 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 18 19:38:03.793812 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788519    2564 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 18 19:38:03.793812 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788523    2564 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 18 19:38:03.793812 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788526    2564 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 18 19:38:03.793812 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788530    2564 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 18 19:38:03.793812 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788533    2564 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 18 19:38:03.793812 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788537    2564 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 18 19:38:03.793812 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788540    2564 flags.go:64] FLAG: --cgroup-root=""
Sep 18 19:38:03.793812 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788543    2564 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 18 19:38:03.794475 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788546    2564 flags.go:64] FLAG: --client-ca-file=""
Sep 18 19:38:03.794475 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788549    2564 flags.go:64] FLAG: --cloud-provider="external"
Sep 18 19:38:03.794475 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788552    2564 flags.go:64] FLAG: --cluster-dns="[]"
Sep 18 19:38:03.794475 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788557    2564 flags.go:64] FLAG: --cluster-domain=""
Sep 18 19:38:03.794475 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788560    2564 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 18 19:38:03.794475 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788563    2564 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 18 19:38:03.794475 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788566    2564 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 18 19:38:03.794475 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788570    2564 flags.go:64] FLAG: --container-log-max-files="5"
Sep 18 19:38:03.794475 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788574    2564 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 18 19:38:03.794475 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788577    2564 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 18 19:38:03.794475 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788580    2564 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 18 19:38:03.794475 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788584    2564 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 18 19:38:03.794475 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788589    2564 flags.go:64] FLAG: --contention-profiling="false"
Sep 18 19:38:03.794475 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788592    2564 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 18 19:38:03.794475 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788596    2564 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 18 19:38:03.794475 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788599    2564 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 18 19:38:03.794475 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788602    2564 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 18 19:38:03.794475 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788606    2564 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 18 19:38:03.794475 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788610    2564 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 18 19:38:03.794475 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788613    2564 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 18 19:38:03.794475 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788616    2564 flags.go:64] FLAG: --enable-load-reader="false"
Sep 18 19:38:03.794475 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788619    2564 flags.go:64] FLAG: --enable-server="true"
Sep 18 19:38:03.794475 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788623    2564 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 18 19:38:03.794475 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788628    2564 flags.go:64] FLAG: --event-burst="100"
Sep 18 19:38:03.794475 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788631    2564 flags.go:64] FLAG: --event-qps="50"
Sep 18 19:38:03.795125 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788634    2564 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 18 19:38:03.795125 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788637    2564 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 18 19:38:03.795125 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788640    2564 flags.go:64] FLAG: --eviction-hard=""
Sep 18 19:38:03.795125 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788644    2564 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 18 19:38:03.795125 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788647    2564 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 18 19:38:03.795125 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788650    2564 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 18 19:38:03.795125 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788653    2564 flags.go:64] FLAG: --eviction-soft=""
Sep 18 19:38:03.795125 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788656    2564 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 18 19:38:03.795125 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788660    2564 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 18 19:38:03.795125 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788662    2564 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 18 19:38:03.795125 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788666    2564 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 18 19:38:03.795125 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788668    2564 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 18 19:38:03.795125 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788672    2564 flags.go:64] FLAG: --fail-swap-on="true"
Sep 18 19:38:03.795125 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788674    2564 flags.go:64] FLAG: --feature-gates=""
Sep 18 19:38:03.795125 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788678    2564 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 18 19:38:03.795125 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788681    2564 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 18 19:38:03.795125 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788684    2564 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 18 19:38:03.795125 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788688    2564 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 18 19:38:03.795125 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788691    2564 flags.go:64] FLAG: --healthz-port="10248"
Sep 18 19:38:03.795125 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788695    2564 flags.go:64] FLAG: --help="false"
Sep 18 19:38:03.795125 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788698    2564 flags.go:64] FLAG: --hostname-override="ip-10-0-128-161.ec2.internal"
Sep 18 19:38:03.795125 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788701    2564 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 18 19:38:03.795125 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788704    2564 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 18 19:38:03.795125 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788707    2564 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 18 19:38:03.795738 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788711    2564 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 18 19:38:03.795738 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788715    2564 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 18 19:38:03.795738 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788718    2564 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 18 19:38:03.795738 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788721    2564 flags.go:64] FLAG: --image-service-endpoint=""
Sep 18 19:38:03.795738 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788724    2564 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 18 19:38:03.795738 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788728    2564 flags.go:64] FLAG: --kube-api-burst="100"
Sep 18 19:38:03.795738 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788731    2564 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 18 19:38:03.795738 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788735    2564 flags.go:64] FLAG: --kube-api-qps="50"
Sep 18 19:38:03.795738 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788738    2564 flags.go:64] FLAG: --kube-reserved=""
Sep 18 19:38:03.795738 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788741    2564 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 18 19:38:03.795738 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788743    2564 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 18 19:38:03.795738 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788747    2564 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 18 19:38:03.795738 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788749    2564 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 18 19:38:03.795738 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788752    2564 flags.go:64] FLAG: --lock-file=""
Sep 18 19:38:03.795738 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788756    2564 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 18 19:38:03.795738 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788759    2564 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 18 19:38:03.795738 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788762    2564 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 18 19:38:03.795738 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788767    2564 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 18 19:38:03.795738 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788770    2564 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 18 19:38:03.795738 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788773    2564 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 18 19:38:03.795738 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788776    2564 flags.go:64] FLAG: --logging-format="text"
Sep 18 19:38:03.795738 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788779    2564 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 18 19:38:03.795738 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788782    2564 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 18 19:38:03.795738 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788785    2564 flags.go:64] FLAG: --manifest-url=""
Sep 18 19:38:03.796343 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788788    2564 flags.go:64] FLAG: --manifest-url-header=""
Sep 18 19:38:03.796343 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788792    2564 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 18 19:38:03.796343 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788795    2564 flags.go:64] FLAG: --max-open-files="1000000"
Sep 18 19:38:03.796343 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788802    2564 flags.go:64] FLAG: --max-pods="110"
Sep 18 19:38:03.796343 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788805    2564 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 18 19:38:03.796343 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788808    2564 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 18 19:38:03.796343 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788811    2564 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 18 19:38:03.796343 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788814    2564 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 18 19:38:03.796343 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788817    2564 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 18 19:38:03.796343 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788821    2564 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 18 19:38:03.796343 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788824    2564 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 18 19:38:03.796343 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788832    2564 flags.go:64] FLAG: --node-status-max-images="50"
Sep 18 19:38:03.796343 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788835    2564 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 18 19:38:03.796343 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788838    2564 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 18 19:38:03.796343 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788841    2564 flags.go:64] FLAG: --pod-cidr=""
Sep 18 19:38:03.796343 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788845    2564 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 18 19:38:03.796343 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788851    2564 flags.go:64] FLAG: --pod-manifest-path=""
Sep 18 19:38:03.796343 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788854    2564 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 18 19:38:03.796343 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788857    2564 flags.go:64] FLAG: --pods-per-core="0"
Sep 18 19:38:03.796343 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788860    2564 flags.go:64] FLAG: --port="10250"
Sep 18 19:38:03.796343 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788863    2564 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 18 19:38:03.796343 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788866    2564 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0eeb8b18b02c22ba7"
Sep 18 19:38:03.796343 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788869    2564 flags.go:64] FLAG: --qos-reserved=""
Sep 18 19:38:03.796343 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788872    2564 flags.go:64] FLAG: --read-only-port="10255"
Sep 18 19:38:03.796961 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788875    2564 flags.go:64] FLAG: --register-node="true"
Sep 18 19:38:03.796961 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788879    2564 flags.go:64] FLAG: --register-with-taints=""
Sep 18 19:38:03.796961 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788883    2564 flags.go:64] FLAG: --registry-burst="10"
Sep 18 19:38:03.796961 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788886    2564 flags.go:64] FLAG: --registry-qps="5"
Sep 18 19:38:03.796961 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788889    2564 flags.go:64] FLAG: --reserved-cpus=""
Sep 18 19:38:03.796961 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788892    2564 flags.go:64] FLAG: --reserved-memory=""
Sep 18 19:38:03.796961 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788895    2564 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 18 19:38:03.796961 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788898    2564 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 18 19:38:03.796961 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788902    2564 flags.go:64] FLAG: --rotate-certificates="false"
Sep 18 19:38:03.796961 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788904    2564 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 18 19:38:03.796961 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788907    2564 flags.go:64] FLAG: --runonce="false"
Sep 18 19:38:03.796961 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788910    2564 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 18 19:38:03.796961 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788914    2564 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 18 19:38:03.796961 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788918    2564 flags.go:64] FLAG: --seccomp-default="false"
Sep 18 19:38:03.796961 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788921    2564 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 18 19:38:03.796961 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788924    2564 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 18 19:38:03.796961 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788927    2564 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 18 19:38:03.796961 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788932    2564 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 18 19:38:03.796961 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788935    2564 flags.go:64] FLAG: --storage-driver-password="root"
Sep 18 19:38:03.796961 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788938    2564 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 18 19:38:03.796961 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788941    2564 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 18 19:38:03.796961 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788944    2564 flags.go:64] FLAG: --storage-driver-user="root"
Sep 18 19:38:03.796961 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788947    2564 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 18 19:38:03.796961 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788950    2564 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 18 19:38:03.796961 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788953    2564 flags.go:64] FLAG: --system-cgroups=""
Sep 18 19:38:03.798086 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788956    2564 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 18 19:38:03.798086 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788962    2564 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 18 19:38:03.798086 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788965    2564 flags.go:64] FLAG: --tls-cert-file=""
Sep 18 19:38:03.798086 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788968    2564 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 18 19:38:03.798086 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788972    2564 flags.go:64] FLAG: --tls-min-version=""
Sep 18 19:38:03.798086 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788975    2564 flags.go:64] FLAG: --tls-private-key-file=""
Sep 18 19:38:03.798086 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788992    2564 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 18 19:38:03.798086 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788995    2564 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 18 19:38:03.798086 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.788999    2564 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 18 19:38:03.798086 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.789002    2564 flags.go:64] FLAG: --v="2"
Sep 18 19:38:03.798086 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.789006    2564 flags.go:64] FLAG: --version="false"
Sep 18 19:38:03.798086 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.789010    2564 flags.go:64] FLAG: --vmodule=""
Sep 18 19:38:03.798086 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.789014    2564 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 18 19:38:03.798086 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.789018    2564 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 18 19:38:03.798086 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789141    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:38:03.798086 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789145    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:38:03.798086 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789148    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:38:03.798086 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789151    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:38:03.798086 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789156    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:38:03.798086 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789161    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:38:03.798086 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789164    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:38:03.798086 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789167    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:38:03.799048 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789170    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:38:03.799048 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789173    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:38:03.799048 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789178    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:38:03.799048 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789181    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:38:03.799048 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789183    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:38:03.799048 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789186    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:38:03.799048 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789189    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:38:03.799048 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789193    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:38:03.799048 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789197    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:38:03.799048 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789200    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:38:03.799048 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789204    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:38:03.799048 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789207    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:38:03.799048 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789210    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:38:03.799048 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789213    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:38:03.799048 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789216    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:38:03.799048 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789220    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:38:03.799048 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789223    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:38:03.799048 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789225    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:38:03.799048 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789228    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:38:03.799878 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789231    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:38:03.799878 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789233    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:38:03.799878 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789236    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:38:03.799878 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789239    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:38:03.799878 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789242    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:38:03.799878 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789245    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:38:03.799878 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789248    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:38:03.799878 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789250    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:38:03.799878 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789253    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:38:03.799878 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789256    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:38:03.799878 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789260    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:38:03.799878 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789263    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:38:03.799878 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789266    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:38:03.799878 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789269    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:38:03.799878 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789271    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:38:03.799878 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789276    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:38:03.799878 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789279    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:38:03.799878 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789282    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:38:03.799878 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789284    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:38:03.800406 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789287    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:38:03.800406 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789291    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:38:03.800406 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789295    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:38:03.800406 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789298    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:38:03.800406 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789301    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:38:03.800406 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789304    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:38:03.800406 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789307    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:38:03.800406 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789310    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:38:03.800406 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789313    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:38:03.800406 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789316    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:38:03.800406 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789319    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:38:03.800406 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789322    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:38:03.800406 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789325    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:38:03.800406 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789327    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:38:03.800406 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789330    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:38:03.800406 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789333    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:38:03.800406 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789335    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:38:03.800406 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789338    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:38:03.800406 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789341    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:38:03.801076 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789343    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:38:03.801076 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789346    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:38:03.801076 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789349    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:38:03.801076 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789352    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:38:03.801076 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789357    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:38:03.801076 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789360    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:38:03.801076 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789363    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:38:03.801076 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789365    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:38:03.801076 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789368    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:38:03.801076 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789372    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:38:03.801076 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789374    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:38:03.801076 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789377    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:38:03.801076 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789380    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:38:03.801076 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789383    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:38:03.801076 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789386    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:38:03.801076 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789389    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:38:03.801076 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789392    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:38:03.801076 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789395    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:38:03.801076 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789398    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:38:03.801076 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789401    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:38:03.801968 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789403    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:38:03.801968 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789406    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:38:03.801968 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789409    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:38:03.801968 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789412    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:38:03.801968 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789415    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:38:03.801968 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789417    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:38:03.801968 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789420    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:38:03.801968 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789422    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:38:03.801968 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789425    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:38:03.801968 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789428    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:38:03.801968 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789430    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:38:03.801968 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789433    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:38:03.801968 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789436    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:38:03.801968 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.789438    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:38:03.801968 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.790059    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 19:38:03.802522 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.799886    2564 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 18 19:38:03.802522 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.799909    2564 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 18 19:38:03.802522 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.800648    2564 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 18 19:38:03.802522 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.800661    2564 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 18 19:38:03.802522 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.800760    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:38:03.802522 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.800770    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:38:03.802522 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.800776    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:38:03.802522 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.800783    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:38:03.802522 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.800790    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:38:03.802522 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.800795    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:38:03.802522 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.800800    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:38:03.802522 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.800805    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:38:03.802522 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.800810    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:38:03.802522 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.800815    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:38:03.802522 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.800819    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:38:03.802522 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.800823    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:38:03.802522 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.800828    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:38:03.802522 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.800832    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:38:03.802522 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.800836    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:38:03.803209 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.800841    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:38:03.803209 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.800845    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:38:03.803209 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.800849    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:38:03.803209 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.800853    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:38:03.803209 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.800858    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:38:03.803209 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.800862    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:38:03.803209 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.800866    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:38:03.803209 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.800870    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:38:03.803209 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.800875    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:38:03.803209 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.800879    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:38:03.803209 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.800884    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:38:03.803209 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.800888    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:38:03.803209 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.800892    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:38:03.803209 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.800896    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:38:03.803209 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.800910    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:38:03.803209 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.800916    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:38:03.803209 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.800921    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:38:03.803209 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.800932    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:38:03.803209 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.800937    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:38:03.803209 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.800941    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:38:03.804090 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.800946    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:38:03.804090 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.800950    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:38:03.804090 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.800954    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:38:03.804090 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.800959    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:38:03.804090 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.800963    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:38:03.804090 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.800967    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:38:03.804090 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.800971    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:38:03.804090 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.800976    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:38:03.804090 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.800994    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:38:03.804090 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.800998    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:38:03.804090 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801003    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:38:03.804090 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801009    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:38:03.804090 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801015    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:38:03.804090 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801020    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:38:03.804090 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801024    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:38:03.804090 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801028    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:38:03.804090 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801034    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:38:03.804090 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801040    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:38:03.804090 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801045    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:38:03.804717 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801050    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:38:03.804717 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801055    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:38:03.804717 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801059    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:38:03.804717 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801063    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:38:03.804717 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801068    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:38:03.804717 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801072    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:38:03.804717 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801076    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:38:03.804717 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801081    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:38:03.804717 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801094    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:38:03.804717 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801098    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:38:03.804717 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801103    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:38:03.804717 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801107    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:38:03.804717 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801116    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:38:03.804717 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801120    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:38:03.804717 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801125    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:38:03.804717 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801129    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:38:03.804717 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801134    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:38:03.804717 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801138    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:38:03.804717 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801142    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:38:03.804717 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801146    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:38:03.805325 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801150    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:38:03.805325 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801154    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:38:03.805325 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801158    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:38:03.805325 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801162    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:38:03.805325 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801167    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:38:03.805325 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801172    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:38:03.805325 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801176    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:38:03.805325 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801180    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:38:03.805325 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801184    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:38:03.805325 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801189    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:38:03.805325 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801193    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:38:03.805325 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801197    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:38:03.805325 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801201    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:38:03.805325 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801206    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:38:03.805325 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801210    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:38:03.805325 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801215    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:38:03.805325 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801219    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:38:03.805325 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801223    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:38:03.805325 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801228    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:38:03.805325 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801232    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:38:03.805847 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801236    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:38:03.805847 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801248    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:38:03.805847 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801253    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:38:03.805847 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801258    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:38:03.805847 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801262    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:38:03.805847 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.801270    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 19:38:03.805847 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801541    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:38:03.805847 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801550    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:38:03.805847 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801555    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:38:03.805847 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801560    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:38:03.805847 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801564    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:38:03.805847 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801569    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:38:03.805847 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801573    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:38:03.805847 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801577    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:38:03.805847 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801581    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:38:03.806270 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801586    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:38:03.806270 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801590    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:38:03.806270 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801594    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:38:03.806270 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801599    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:38:03.806270 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801603    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:38:03.806270 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801607    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:38:03.806270 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801611    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:38:03.806270 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801617    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:38:03.806270 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801621    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:38:03.806270 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801625    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:38:03.806270 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801630    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:38:03.806270 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801635    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:38:03.806270 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801639    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:38:03.806270 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801643    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:38:03.806270 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801648    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:38:03.806270 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801652    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:38:03.806270 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801656    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:38:03.806270 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801660    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:38:03.806270 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801664    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:38:03.806750 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801678    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:38:03.806750 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801683    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:38:03.806750 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801687    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:38:03.806750 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801691    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:38:03.806750 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801696    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:38:03.806750 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801700    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:38:03.806750 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801704    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:38:03.806750 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801708    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:38:03.806750 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801712    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:38:03.806750 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801717    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:38:03.806750 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801721    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:38:03.806750 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801725    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:38:03.806750 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801729    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:38:03.806750 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801735    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:38:03.806750 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801741    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:38:03.806750 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801746    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:38:03.806750 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801752    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:38:03.806750 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801758    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:38:03.806750 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801762    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:38:03.807227 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801767    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:38:03.807227 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801773    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:38:03.807227 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801778    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:38:03.807227 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801782    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:38:03.807227 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801787    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:38:03.807227 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801791    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:38:03.807227 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801796    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:38:03.807227 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801800    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:38:03.807227 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801805    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:38:03.807227 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801809    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:38:03.807227 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801814    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:38:03.807227 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801818    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:38:03.807227 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801823    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:38:03.807227 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801827    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:38:03.807227 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801840    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:38:03.807227 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801847    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:38:03.807227 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801853    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:38:03.807227 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801857    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:38:03.807227 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801862    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:38:03.807227 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801866    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:38:03.807720 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801870    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:38:03.807720 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801874    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:38:03.807720 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801878    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:38:03.807720 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801882    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:38:03.807720 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801886    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:38:03.807720 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801890    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:38:03.807720 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801896    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:38:03.807720 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801900    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:38:03.807720 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801904    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:38:03.807720 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801908    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:38:03.807720 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801913    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:38:03.807720 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801917    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:38:03.807720 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801921    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:38:03.807720 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801926    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:38:03.807720 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801930    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:38:03.807720 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801934    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:38:03.807720 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801938    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:38:03.807720 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801942    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:38:03.807720 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801947    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:38:03.807720 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801951    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:38:03.808225 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801955    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:38:03.808225 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801959    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:38:03.808225 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801964    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:38:03.808225 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801968    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:38:03.808225 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801972    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:38:03.808225 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.801976    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:38:03.808225 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.802002    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:38:03.808225 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.802015    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:38:03.808225 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.802019    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:38:03.808225 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.802023    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:38:03.808225 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.802028    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:38:03.808225 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:03.802032    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:38:03.808225 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.802040    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 19:38:03.808225 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.802233    2564 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 18 19:38:03.808582 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.805077    2564 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 18 19:38:03.808582 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.806009    2564 server.go:1019] "Starting client certificate rotation"
Sep 18 19:38:03.808582 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.806116    2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 19:38:03.808582 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.806802    2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 19:38:03.812092 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.812064    2564 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 19:38:03.814337 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.814315    2564 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 19:38:03.825684 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.825665    2564 log.go:25] "Validated CRI v1 runtime API"
Sep 18 19:38:03.827379 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.827364    2564 log.go:25] "Validated CRI v1 image API"
Sep 18 19:38:03.828209 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.828194    2564 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 18 19:38:03.830617 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.830598    2564 fs.go:135] Filesystem UUIDs: map[55aad574-4825-4b8c-b69e-5ea84852cc23:/dev/nvme0n1p4 66ac1999-7e5f-4ee5-9b20-5fe5096b9a2e:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Sep 18 19:38:03.830673 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.830617    2564 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 18 19:38:03.835863 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.835747    2564 manager.go:216] Machine: {Timestamp:2026-09-18 19:38:03.834391409 +0000 UTC m=+0.344747512 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100008 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25a8d073cd737e9d783341d3977b41 SystemUUID:ec25a8d0-73cd-737e-9d78-3341d3977b41 BootID:dd9826fe-4dc9-44f8-8351-f1079daa7bd3 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ec:a9:f5:13 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ec:a9:f5:13 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:de:92:54:26:a3:5b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 18 19:38:03.835863 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.835859    2564 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 18 19:38:03.836031 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.835945    2564 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 18 19:38:03.839422 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.839392    2564 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 18 19:38:03.839576 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.839427    2564 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-161.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 18 19:38:03.840763 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.840752    2564 topology_manager.go:138] "Creating topology manager with none policy"
Sep 18 19:38:03.840795 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.840765    2564 container_manager_linux.go:309] "Creating device plugin manager"
Sep 18 19:38:03.840795 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.840773    2564 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 19:38:03.841405 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.841395    2564 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 19:38:03.842727 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.842716    2564 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 18 19:38:03.845540 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.845527    2564 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 19:38:03.845678 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.845661    2564 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 18 19:38:03.845722 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.845690    2564 kubelet.go:486] "Attempting to sync node with API server"
Sep 18 19:38:03.845722 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.845703    2564 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 18 19:38:03.845722 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.845717    2564 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 18 19:38:03.845840 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.845727    2564 kubelet.go:398] "Adding apiserver pod source"
Sep 18 19:38:03.845840 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.845736    2564 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 18 19:38:03.846274 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.846260    2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 19:38:03.846319 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.846289    2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 19:38:03.848028 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.847992    2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 19:38:03.849024 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.849009    2564 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 18 19:38:03.852913 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.852892    2564 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 18 19:38:03.853000 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.852945    2564 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 18 19:38:03.853952 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.853938    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 18 19:38:03.853952 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.853955    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 18 19:38:03.854054 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.853961    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 18 19:38:03.854054 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.853967    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 18 19:38:03.854054 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.853975    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 18 19:38:03.854054 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.853996    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 18 19:38:03.854054 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.854010    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 18 19:38:03.854054 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.854016    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 18 19:38:03.854054 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.854023    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 18 19:38:03.854054 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.854032    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 18 19:38:03.854054 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.854045    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 18 19:38:03.854054 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.854054    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 18 19:38:03.854851 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.854837    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 18 19:38:03.854885 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.854852    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 18 19:38:03.856560 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.856542    2564 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-161.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 19:38:03.856632 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:03.856544    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 18 19:38:03.856632 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:03.856537    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-161.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 18 19:38:03.858620 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.858606    2564 server.go:1268] "Started kubelet"
Sep 18 19:38:03.858699 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.858671    2564 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 18 19:38:03.858807 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.858782    2564 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 18 19:38:03.858879 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.858827    2564 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 18 19:38:03.859749 ip-10-0-128-161 systemd[1]: Started Kubernetes Kubelet.
Sep 18 19:38:03.860950 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.860750    2564 server.go:310] "Adding debug handlers to kubelet server"
Sep 18 19:38:03.860950 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.860916    2564 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 18 19:38:03.863969 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:03.863950    2564 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 18 19:38:03.866362 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:03.865495    2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-161.ec2.internal.18d6817ee07e1a2a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-161.ec2.internal,UID:ip-10-0-128-161.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-161.ec2.internal,},FirstTimestamp:2026-09-18 19:38:03.858573866 +0000 UTC m=+0.368929985,LastTimestamp:2026-09-18 19:38:03.858573866 +0000 UTC m=+0.368929985,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-161.ec2.internal,}"
Sep 18 19:38:03.867557 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.867539    2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 18 19:38:03.868165 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.868151    2564 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 18 19:38:03.868262 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:03.868242    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-161.ec2.internal\" not found"
Sep 18 19:38:03.868669 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.868654    2564 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 18 19:38:03.868742 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.868697    2564 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 18 19:38:03.868742 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.868710    2564 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 18 19:38:03.868742 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.868719    2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fvzx9"
Sep 18 19:38:03.868868 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.868813    2564 reconstruct.go:97] "Volume reconstruction finished"
Sep 18 19:38:03.868868 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.868821    2564 reconciler.go:29] "Reconciler: start to sync state"
Sep 18 19:38:03.869165 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.869146    2564 factory.go:55] Registering systemd factory
Sep 18 19:38:03.869242 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.869184    2564 factory.go:223] Registration of the systemd container factory successfully
Sep 18 19:38:03.869417 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.869403    2564 factory.go:153] Registering CRI-O factory
Sep 18 19:38:03.869471 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.869421    2564 factory.go:223] Registration of the crio container factory successfully
Sep 18 19:38:03.869529 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.869477    2564 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 18 19:38:03.869529 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.869497    2564 factory.go:103] Registering Raw factory
Sep 18 19:38:03.869529 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.869516    2564 manager.go:1196] Started watching for new ooms in manager
Sep 18 19:38:03.869705 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:03.869657    2564 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-161.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 18 19:38:03.869772 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:03.869746    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 18 19:38:03.869951 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.869939    2564 manager.go:349] Starting recovery of all containers
Sep 18 19:38:03.876047 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.875944    2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fvzx9"
Sep 18 19:38:03.882333 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.882305    2564 manager.go:354] Recovery completed
Sep 18 19:38:03.888751 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.888616    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:38:03.890807 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.890786    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-161.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:38:03.890886 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.890823    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-161.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:38:03.890886 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.890836    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-161.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:38:03.891552 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.891527    2564 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 18 19:38:03.891552 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.891543    2564 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 18 19:38:03.891698 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.891561    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:38:03.891698 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.891562    2564 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 19:38:03.892280 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.892264    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-161.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:38:03.892346 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.892296    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-161.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:38:03.892346 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.892312    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-161.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:38:03.892794 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.892781    2564 policy_none.go:49] "None policy: Start"
Sep 18 19:38:03.892852 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.892799    2564 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 18 19:38:03.892852 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.892813    2564 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 18 19:38:03.894136 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.894124    2564 policy_none.go:47] "Start"
Sep 18 19:38:03.929643 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.929625    2564 manager.go:337] "Starting Device Plugin manager"
Sep 18 19:38:03.946823 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:03.929666    2564 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 18 19:38:03.946823 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.929677    2564 server.go:87] "Starting device plugin registration server"
Sep 18 19:38:03.946823 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.930078    2564 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 18 19:38:03.946823 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.930112    2564 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 18 19:38:03.946823 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.930195    2564 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 18 19:38:03.946823 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.930301    2564 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 18 19:38:03.946823 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.930312    2564 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 18 19:38:03.946823 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:03.931880    2564 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 18 19:38:03.946823 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:03.931959    2564 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-161.ec2.internal\" not found"
Sep 18 19:38:03.991802 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.991769    2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 18 19:38:03.993069 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.993042    2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 18 19:38:03.993185 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.993101    2564 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 18 19:38:03.993185 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.993131    2564 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 18 19:38:03.993281 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:03.993244    2564 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 18 19:38:03.995551 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:03.995533    2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:38:04.030416 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.030385    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:38:04.032249 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.032155    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-161.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:38:04.032249 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.032188    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-161.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:38:04.032249 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.032225    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-161.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:38:04.032249 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.032249    2564 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-161.ec2.internal"
Sep 18 19:38:04.040407 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.040386    2564 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-161.ec2.internal"
Sep 18 19:38:04.040528 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:04.040412    2564 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-161.ec2.internal\": node \"ip-10-0-128-161.ec2.internal\" not found"
Sep 18 19:38:04.055866 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:04.055841    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-161.ec2.internal\" not found"
Sep 18 19:38:04.093562 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.093529    2564 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-161.ec2.internal"]
Sep 18 19:38:04.093711 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.093608    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:38:04.094662 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.094645    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-161.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:38:04.094761 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.094677    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-161.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:38:04.094761 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.094686    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-161.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:38:04.096089 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.096077    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:38:04.096207 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.096193    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.ec2.internal"
Sep 18 19:38:04.096241 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.096222    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:38:04.096884 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.096862    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-161.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:38:04.096955 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.096898    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-161.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:38:04.096955 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.096907    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-161.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:38:04.097464 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.097450    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-161.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:38:04.097524 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.097475    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-161.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:38:04.097524 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.097489    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-161.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:38:04.098937 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.098923    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-161.ec2.internal"
Sep 18 19:38:04.098998 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.098948    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:38:04.099616 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.099601    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-161.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:38:04.099673 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.099640    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-161.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:38:04.099673 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.099652    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-161.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:38:04.115625 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:04.115604    2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-161.ec2.internal\" not found" node="ip-10-0-128-161.ec2.internal"
Sep 18 19:38:04.119742 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:04.119726    2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-161.ec2.internal\" not found" node="ip-10-0-128-161.ec2.internal"
Sep 18 19:38:04.156046 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:04.156021    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-161.ec2.internal\" not found"
Sep 18 19:38:04.256840 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:04.256792    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-161.ec2.internal\" not found"
Sep 18 19:38:04.270137 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.270099    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c1e9c7cacf21e14413fce157c96f3cd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-161.ec2.internal\" (UID: \"2c1e9c7cacf21e14413fce157c96f3cd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.ec2.internal"
Sep 18 19:38:04.270137 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.270141    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9eb8c7aafdc391f2f7ae0f04a98530ea-config\") pod \"kube-apiserver-proxy-ip-10-0-128-161.ec2.internal\" (UID: \"9eb8c7aafdc391f2f7ae0f04a98530ea\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-161.ec2.internal"
Sep 18 19:38:04.270282 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.270166    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2c1e9c7cacf21e14413fce157c96f3cd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-161.ec2.internal\" (UID: \"2c1e9c7cacf21e14413fce157c96f3cd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.ec2.internal"
Sep 18 19:38:04.357581 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:04.357510    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-161.ec2.internal\" not found"
Sep 18 19:38:04.370931 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.370894    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9eb8c7aafdc391f2f7ae0f04a98530ea-config\") pod \"kube-apiserver-proxy-ip-10-0-128-161.ec2.internal\" (UID: \"9eb8c7aafdc391f2f7ae0f04a98530ea\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-161.ec2.internal"
Sep 18 19:38:04.370931 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.370933    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2c1e9c7cacf21e14413fce157c96f3cd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-161.ec2.internal\" (UID: \"2c1e9c7cacf21e14413fce157c96f3cd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.ec2.internal"
Sep 18 19:38:04.371078 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.370951    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c1e9c7cacf21e14413fce157c96f3cd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-161.ec2.internal\" (UID: \"2c1e9c7cacf21e14413fce157c96f3cd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.ec2.internal"
Sep 18 19:38:04.371078 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.371005    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2c1e9c7cacf21e14413fce157c96f3cd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-161.ec2.internal\" (UID: \"2c1e9c7cacf21e14413fce157c96f3cd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.ec2.internal"
Sep 18 19:38:04.371078 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.371010    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9eb8c7aafdc391f2f7ae0f04a98530ea-config\") pod \"kube-apiserver-proxy-ip-10-0-128-161.ec2.internal\" (UID: \"9eb8c7aafdc391f2f7ae0f04a98530ea\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-161.ec2.internal"
Sep 18 19:38:04.371078 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.371030    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c1e9c7cacf21e14413fce157c96f3cd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-161.ec2.internal\" (UID: \"2c1e9c7cacf21e14413fce157c96f3cd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.ec2.internal"
Sep 18 19:38:04.420058 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.420028    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.ec2.internal"
Sep 18 19:38:04.424479 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.424408    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-161.ec2.internal"
Sep 18 19:38:04.458323 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:04.458291    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-161.ec2.internal\" not found"
Sep 18 19:38:04.558851 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:04.558804    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-161.ec2.internal\" not found"
Sep 18 19:38:04.659371 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:04.659297    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-161.ec2.internal\" not found"
Sep 18 19:38:04.739376 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.739349    2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:38:04.768785 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.768754    2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-161.ec2.internal"
Sep 18 19:38:04.778050 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.778027    2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 19:38:04.780937 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.780915    2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.ec2.internal"
Sep 18 19:38:04.789825 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.789806    2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 19:38:04.805996 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.805959    2564 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 18 19:38:04.806138 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.806117    2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 19:38:04.806179 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.806158    2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 19:38:04.846287 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.846255    2564 apiserver.go:52] "Watching apiserver"
Sep 18 19:38:04.862143 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.862110    2564 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 18 19:38:04.862693 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.862671    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-dwwf5","openshift-image-registry/node-ca-8hdsx","openshift-multus/network-metrics-daemon-xgxbn","openshift-network-diagnostics/network-check-target-857bd","openshift-ovn-kubernetes/ovnkube-node-xcbfz","kube-system/global-pull-secret-syncer-6vdpn","kube-system/konnectivity-agent-667km","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.ec2.internal","openshift-multus/multus-additional-cni-plugins-szj4k","openshift-multus/multus-vxbw8","openshift-network-operator/iptables-alerter-tm2rm","kube-system/kube-apiserver-proxy-ip-10-0-128-161.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47hjk","openshift-cluster-node-tuning-operator/tuned-96v5g"]
Sep 18 19:38:04.865141 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.865119    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-667km"
Sep 18 19:38:04.867892 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.867517    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 18 19:38:04.867892 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.867521    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 18 19:38:04.867892 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.867625    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-j4xl2\""
Sep 18 19:38:04.867892 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.867641    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47hjk"
Sep 18 19:38:04.867892 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.867840    2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 18 19:38:04.870878 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.870374    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 18 19:38:04.870878 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.870628    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 18 19:38:04.870878 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.870685    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 18 19:38:04.870878 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.870750    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zh2j5\""
Sep 18 19:38:04.871157 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.870958    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xgxbn"
Sep 18 19:38:04.871157 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:04.871081    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xgxbn" podUID="256a72d8-e079-42fb-9426-c53adc283d1f"
Sep 18 19:38:04.873485 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.873464    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ecb576b0-eafa-4997-9690-36f2ac243783-agent-certs\") pod \"konnectivity-agent-667km\" (UID: \"ecb576b0-eafa-4997-9690-36f2ac243783\") " pod="kube-system/konnectivity-agent-667km"
Sep 18 19:38:04.873598 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.873489    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f6d3d568-a824-4226-9944-66adc0973032-kubelet-dir\") pod \"aws-ebs-csi-driver-node-47hjk\" (UID: \"f6d3d568-a824-4226-9944-66adc0973032\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47hjk"
Sep 18 19:38:04.873598 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.873507    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f6d3d568-a824-4226-9944-66adc0973032-socket-dir\") pod \"aws-ebs-csi-driver-node-47hjk\" (UID: \"f6d3d568-a824-4226-9944-66adc0973032\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47hjk"
Sep 18 19:38:04.873598 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.873522    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f6d3d568-a824-4226-9944-66adc0973032-registration-dir\") pod \"aws-ebs-csi-driver-node-47hjk\" (UID: \"f6d3d568-a824-4226-9944-66adc0973032\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47hjk"
Sep 18 19:38:04.873598 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.873537    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-857bd"
Sep 18 19:38:04.873775 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.873539    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/256a72d8-e079-42fb-9426-c53adc283d1f-metrics-certs\") pod \"network-metrics-daemon-xgxbn\" (UID: \"256a72d8-e079-42fb-9426-c53adc283d1f\") " pod="openshift-multus/network-metrics-daemon-xgxbn"
Sep 18 19:38:04.873775 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:04.873610    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-857bd" podUID="1b42bdad-4122-4a0a-978c-d2d5a7bdbd55"
Sep 18 19:38:04.873775 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.873633    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ecb576b0-eafa-4997-9690-36f2ac243783-konnectivity-ca\") pod \"konnectivity-agent-667km\" (UID: \"ecb576b0-eafa-4997-9690-36f2ac243783\") " pod="kube-system/konnectivity-agent-667km"
Sep 18 19:38:04.873775 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.873647    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f6d3d568-a824-4226-9944-66adc0973032-device-dir\") pod \"aws-ebs-csi-driver-node-47hjk\" (UID: \"f6d3d568-a824-4226-9944-66adc0973032\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47hjk"
Sep 18 19:38:04.873775 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.873661    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f6d3d568-a824-4226-9944-66adc0973032-etc-selinux\") pod \"aws-ebs-csi-driver-node-47hjk\" (UID: \"f6d3d568-a824-4226-9944-66adc0973032\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47hjk"
Sep 18 19:38:04.873775 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.873674    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f6d3d568-a824-4226-9944-66adc0973032-sys-fs\") pod \"aws-ebs-csi-driver-node-47hjk\" (UID: \"f6d3d568-a824-4226-9944-66adc0973032\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47hjk"
Sep 18 19:38:04.873775 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.873694    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzztj\" (UniqueName: \"kubernetes.io/projected/f6d3d568-a824-4226-9944-66adc0973032-kube-api-access-xzztj\") pod \"aws-ebs-csi-driver-node-47hjk\" (UID: \"f6d3d568-a824-4226-9944-66adc0973032\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47hjk"
Sep 18 19:38:04.873775 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.873737    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwfbj\" (UniqueName: \"kubernetes.io/projected/256a72d8-e079-42fb-9426-c53adc283d1f-kube-api-access-wwfbj\") pod \"network-metrics-daemon-xgxbn\" (UID: \"256a72d8-e079-42fb-9426-c53adc283d1f\") " pod="openshift-multus/network-metrics-daemon-xgxbn"
Sep 18 19:38:04.875609 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.875588    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz"
Sep 18 19:38:04.877621 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.877571    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 18 19:38:04.877726 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.877616    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 18 19:38:04.877726 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.877647    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 18 19:38:04.877726 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.877698    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 18 19:38:04.877876 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.877616    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 18 19:38:04.877876 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.877853    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 18 19:38:04.878211 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.878198    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hmlj7\""
Sep 18 19:38:04.878417 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.878400    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vdpn"
Sep 18 19:38:04.878500 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:04.878474    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vdpn" podUID="2939ca32-81a7-4e9d-bdc6-ffca73820918"
Sep 18 19:38:04.879435 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.879416    2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 19:38:04.880922 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.880905    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-szj4k"
Sep 18 19:38:04.882770 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.882752    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 18 19:38:04.882864 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.882821    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jnq68\""
Sep 18 19:38:04.882864 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.882827    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 18 19:38:04.883094 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.883076    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 18 19:38:04.883197 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.883126    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 18 19:38:04.883197 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.883155    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 18 19:38:04.883498 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.883484    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dwwf5"
Sep 18 19:38:04.885090 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.885064    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-17 19:33:04 +0000 UTC" deadline="2028-05-29 00:35:57.61242419 +0000 UTC"
Sep 18 19:38:04.885090 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.885087    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14836h57m52.727339959s"
Sep 18 19:38:04.885209 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.885190    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8mlf2\""
Sep 18 19:38:04.885467 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.885454    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 18 19:38:04.885503 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.885487    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 18 19:38:04.886533 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.886516    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8hdsx"
Sep 18 19:38:04.888272 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.888250    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 18 19:38:04.888378 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.888323    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-27rzb\""
Sep 18 19:38:04.888378 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.888332    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 18 19:38:04.888465 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.888402    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 18 19:38:04.888820 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.888803    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vxbw8"
Sep 18 19:38:04.890575 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.890550    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 18 19:38:04.890655 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.890619    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-4bh2j\""
Sep 18 19:38:04.891024 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.891009    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tm2rm"
Sep 18 19:38:04.892514 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.892497    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 18 19:38:04.893005 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.892963    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 18 19:38:04.893095 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.893004    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 18 19:38:04.893095 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.893022    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6cfz6\""
Sep 18 19:38:04.894788 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.894768    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-96v5g"
Sep 18 19:38:04.896667 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.896635    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 18 19:38:04.896667 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.896664    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 18 19:38:04.896799 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.896742    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vvjv9\""
Sep 18 19:38:04.902003 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.901928    2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-lg6nj"
Sep 18 19:38:04.909028 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.908999    2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-lg6nj"
Sep 18 19:38:04.954353 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:04.954318    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eb8c7aafdc391f2f7ae0f04a98530ea.slice/crio-ba8af836b80c3721c9fdd45b677e19350b041c34e11be86736e86d3bc773cffd WatchSource:0}: Error finding container ba8af836b80c3721c9fdd45b677e19350b041c34e11be86736e86d3bc773cffd: Status 404 returned error can't find the container with id ba8af836b80c3721c9fdd45b677e19350b041c34e11be86736e86d3bc773cffd
Sep 18 19:38:04.954623 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:04.954594    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c1e9c7cacf21e14413fce157c96f3cd.slice/crio-0334d70254f30993173e3d99e3accfb83aa287555694934633ebeec541a62a49 WatchSource:0}: Error finding container 0334d70254f30993173e3d99e3accfb83aa287555694934633ebeec541a62a49: Status 404 returned error can't find the container with id 0334d70254f30993173e3d99e3accfb83aa287555694934633ebeec541a62a49
Sep 18 19:38:04.959352 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.959323    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:38:04.970054 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.970031    2564 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 18 19:38:04.974254 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.974222    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/db9632d7-9a01-4dc2-a1ac-e1e7505e7b62-lib-modules\") pod \"tuned-96v5g\" (UID: \"db9632d7-9a01-4dc2-a1ac-e1e7505e7b62\") " pod="openshift-cluster-node-tuning-operator/tuned-96v5g"
Sep 18 19:38:04.974340 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.974262    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjgc5\" (UniqueName: \"kubernetes.io/projected/1b42bdad-4122-4a0a-978c-d2d5a7bdbd55-kube-api-access-zjgc5\") pod \"network-check-target-857bd\" (UID: \"1b42bdad-4122-4a0a-978c-d2d5a7bdbd55\") " pod="openshift-network-diagnostics/network-check-target-857bd"
Sep 18 19:38:04.974340 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.974287    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2fd6f85b-2fd5-456a-919c-ca59d7e5203f-systemd-units\") pod \"ovnkube-node-xcbfz\" (UID: \"2fd6f85b-2fd5-456a-919c-ca59d7e5203f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz"
Sep 18 19:38:04.974340 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.974312    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2b852eeb-95f7-40df-adbf-341896362824-host-run-netns\") pod \"multus-vxbw8\" (UID: \"2b852eeb-95f7-40df-adbf-341896362824\") " pod="openshift-multus/multus-vxbw8"
Sep 18 19:38:04.974340 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.974327    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2b852eeb-95f7-40df-adbf-341896362824-hostroot\") pod \"multus-vxbw8\" (UID: \"2b852eeb-95f7-40df-adbf-341896362824\") " pod="openshift-multus/multus-vxbw8"
Sep 18 19:38:04.974461 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.974341    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2fd6f85b-2fd5-456a-919c-ca59d7e5203f-host-slash\") pod \"ovnkube-node-xcbfz\" (UID: \"2fd6f85b-2fd5-456a-919c-ca59d7e5203f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz"
Sep 18 19:38:04.974461 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.974354    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2fd6f85b-2fd5-456a-919c-ca59d7e5203f-host-cni-bin\") pod \"ovnkube-node-xcbfz\" (UID: \"2fd6f85b-2fd5-456a-919c-ca59d7e5203f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz"
Sep 18 19:38:04.974461 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.974371    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2fd6f85b-2fd5-456a-919c-ca59d7e5203f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xcbfz\" (UID: \"2fd6f85b-2fd5-456a-919c-ca59d7e5203f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz"
Sep 18 19:38:04.974461 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.974407    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2fd6f85b-2fd5-456a-919c-ca59d7e5203f-ovn-node-metrics-cert\") pod \"ovnkube-node-xcbfz\" (UID: \"2fd6f85b-2fd5-456a-919c-ca59d7e5203f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz"
Sep 18 19:38:04.974461 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.974446    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ecb576b0-eafa-4997-9690-36f2ac243783-konnectivity-ca\") pod \"konnectivity-agent-667km\" (UID: \"ecb576b0-eafa-4997-9690-36f2ac243783\") " pod="kube-system/konnectivity-agent-667km"
Sep 18 19:38:04.974614 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.974538    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f6d3d568-a824-4226-9944-66adc0973032-device-dir\") pod \"aws-ebs-csi-driver-node-47hjk\" (UID: \"f6d3d568-a824-4226-9944-66adc0973032\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47hjk"
Sep 18 19:38:04.974614 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.974577    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2939ca32-81a7-4e9d-bdc6-ffca73820918-original-pull-secret\") pod \"global-pull-secret-syncer-6vdpn\" (UID: \"2939ca32-81a7-4e9d-bdc6-ffca73820918\") " pod="kube-system/global-pull-secret-syncer-6vdpn"
Sep 18 19:38:04.974672 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.974611    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6df88db0-10f0-429d-b36f-644069df51ed-system-cni-dir\") pod \"multus-additional-cni-plugins-szj4k\" (UID: \"6df88db0-10f0-429d-b36f-644069df51ed\") " pod="openshift-multus/multus-additional-cni-plugins-szj4k"
Sep 18 19:38:04.974672 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.974656    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5m8v\" (UniqueName: \"kubernetes.io/projected/6df88db0-10f0-429d-b36f-644069df51ed-kube-api-access-p5m8v\") pod \"multus-additional-cni-plugins-szj4k\" (UID: \"6df88db0-10f0-429d-b36f-644069df51ed\") " pod="openshift-multus/multus-additional-cni-plugins-szj4k"
Sep 18 19:38:04.974728 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.974691    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/937fca6a-d8cf-48b9-a0e5-39572043858e-iptables-alerter-script\") pod \"iptables-alerter-tm2rm\" (UID: \"937fca6a-d8cf-48b9-a0e5-39572043858e\") " pod="openshift-network-operator/iptables-alerter-tm2rm"
Sep 18 19:38:04.974728 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.974705    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f6d3d568-a824-4226-9944-66adc0973032-device-dir\") pod \"aws-ebs-csi-driver-node-47hjk\" (UID: \"f6d3d568-a824-4226-9944-66adc0973032\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47hjk"
Sep 18 19:38:04.974819 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.974767    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/db9632d7-9a01-4dc2-a1ac-e1e7505e7b62-etc-sysctl-conf\") pod \"tuned-96v5g\" (UID: \"db9632d7-9a01-4dc2-a1ac-e1e7505e7b62\") " pod="openshift-cluster-node-tuning-operator/tuned-96v5g"
Sep 18 19:38:04.974819 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.974793    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/db9632d7-9a01-4dc2-a1ac-e1e7505e7b62-run\") pod \"tuned-96v5g\" (UID: \"db9632d7-9a01-4dc2-a1ac-e1e7505e7b62\") " pod="openshift-cluster-node-tuning-operator/tuned-96v5g"
Sep 18 19:38:04.974961 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.974823    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f6d3d568-a824-4226-9944-66adc0973032-socket-dir\") pod \"aws-ebs-csi-driver-node-47hjk\" (UID: \"f6d3d568-a824-4226-9944-66adc0973032\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47hjk"
Sep 18 19:38:04.974961 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.974854    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2939ca32-81a7-4e9d-bdc6-ffca73820918-dbus\") pod \"global-pull-secret-syncer-6vdpn\" (UID: \"2939ca32-81a7-4e9d-bdc6-ffca73820918\") " pod="kube-system/global-pull-secret-syncer-6vdpn"
Sep 18 19:38:04.974961 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.974925    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2b852eeb-95f7-40df-adbf-341896362824-host-var-lib-cni-multus\") pod \"multus-vxbw8\" (UID: \"2b852eeb-95f7-40df-adbf-341896362824\") " pod="openshift-multus/multus-vxbw8"
Sep 18 19:38:04.974961 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.974941    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ecb576b0-eafa-4997-9690-36f2ac243783-konnectivity-ca\") pod \"konnectivity-agent-667km\" (UID: \"ecb576b0-eafa-4997-9690-36f2ac243783\") " pod="kube-system/konnectivity-agent-667km"
Sep 18 19:38:04.974961 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.974958    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f6d3d568-a824-4226-9944-66adc0973032-socket-dir\") pod \"aws-ebs-csi-driver-node-47hjk\" (UID: \"f6d3d568-a824-4226-9944-66adc0973032\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47hjk"
Sep 18 19:38:04.975214 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.974961    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2b852eeb-95f7-40df-adbf-341896362824-multus-daemon-config\") pod \"multus-vxbw8\" (UID: \"2b852eeb-95f7-40df-adbf-341896362824\") " pod="openshift-multus/multus-vxbw8"
Sep 18 19:38:04.975214 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.975016    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/db9632d7-9a01-4dc2-a1ac-e1e7505e7b62-sys\") pod \"tuned-96v5g\" (UID: \"db9632d7-9a01-4dc2-a1ac-e1e7505e7b62\") " pod="openshift-cluster-node-tuning-operator/tuned-96v5g"
Sep 18 19:38:04.975214 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.975038    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/db9632d7-9a01-4dc2-a1ac-e1e7505e7b62-host\") pod \"tuned-96v5g\" (UID: \"db9632d7-9a01-4dc2-a1ac-e1e7505e7b62\") " pod="openshift-cluster-node-tuning-operator/tuned-96v5g"
Sep 18 19:38:04.975214 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.975055    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2fd6f85b-2fd5-456a-919c-ca59d7e5203f-host-kubelet\") pod \"ovnkube-node-xcbfz\" (UID: \"2fd6f85b-2fd5-456a-919c-ca59d7e5203f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz"
Sep 18 19:38:04.975214 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.975075    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b852eeb-95f7-40df-adbf-341896362824-system-cni-dir\") pod \"multus-vxbw8\" (UID: \"2b852eeb-95f7-40df-adbf-341896362824\") " pod="openshift-multus/multus-vxbw8"
Sep 18 19:38:04.975214 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.975097    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2fd6f85b-2fd5-456a-919c-ca59d7e5203f-ovnkube-config\") pod \"ovnkube-node-xcbfz\" (UID: \"2fd6f85b-2fd5-456a-919c-ca59d7e5203f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz"
Sep 18 19:38:04.975214 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.975122    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwfbj\" (UniqueName: \"kubernetes.io/projected/256a72d8-e079-42fb-9426-c53adc283d1f-kube-api-access-wwfbj\") pod \"network-metrics-daemon-xgxbn\" (UID: \"256a72d8-e079-42fb-9426-c53adc283d1f\") " pod="openshift-multus/network-metrics-daemon-xgxbn"
Sep 18 19:38:04.975214 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.975141    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/256a72d8-e079-42fb-9426-c53adc283d1f-metrics-certs\") pod \"network-metrics-daemon-xgxbn\" (UID: \"256a72d8-e079-42fb-9426-c53adc283d1f\") " pod="openshift-multus/network-metrics-daemon-xgxbn"
Sep 18 19:38:04.975214 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.975155    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/db9632d7-9a01-4dc2-a1ac-e1e7505e7b62-etc-systemd\") pod \"tuned-96v5g\" (UID: \"db9632d7-9a01-4dc2-a1ac-e1e7505e7b62\") " pod="openshift-cluster-node-tuning-operator/tuned-96v5g"
Sep 18 19:38:04.975548 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:04.975227    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:04.975548 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.975258    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q2lc\" (UniqueName: \"kubernetes.io/projected/db9632d7-9a01-4dc2-a1ac-e1e7505e7b62-kube-api-access-2q2lc\") pod \"tuned-96v5g\" (UID: \"db9632d7-9a01-4dc2-a1ac-e1e7505e7b62\") " pod="openshift-cluster-node-tuning-operator/tuned-96v5g"
Sep 18 19:38:04.975548 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:04.975285    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/256a72d8-e079-42fb-9426-c53adc283d1f-metrics-certs podName:256a72d8-e079-42fb-9426-c53adc283d1f nodeName:}" failed. No retries permitted until 2026-09-18 19:38:05.475258027 +0000 UTC m=+1.985614119 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/256a72d8-e079-42fb-9426-c53adc283d1f-metrics-certs") pod "network-metrics-daemon-xgxbn" (UID: "256a72d8-e079-42fb-9426-c53adc283d1f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:04.975548 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.975308    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2fd6f85b-2fd5-456a-919c-ca59d7e5203f-host-run-ovn-kubernetes\") pod \"ovnkube-node-xcbfz\" (UID: \"2fd6f85b-2fd5-456a-919c-ca59d7e5203f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz"
Sep 18 19:38:04.975548 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.975330    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2fd6f85b-2fd5-456a-919c-ca59d7e5203f-env-overrides\") pod \"ovnkube-node-xcbfz\" (UID: \"2fd6f85b-2fd5-456a-919c-ca59d7e5203f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz"
Sep 18 19:38:04.975548 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.975346    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f6d3d568-a824-4226-9944-66adc0973032-etc-selinux\") pod \"aws-ebs-csi-driver-node-47hjk\" (UID: \"f6d3d568-a824-4226-9944-66adc0973032\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47hjk"
Sep 18 19:38:04.975548 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.975361    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6df88db0-10f0-429d-b36f-644069df51ed-tuning-conf-dir\") pod \"multus-additional-cni-plugins-szj4k\" (UID: \"6df88db0-10f0-429d-b36f-644069df51ed\") " pod="openshift-multus/multus-additional-cni-plugins-szj4k"
Sep 18 19:38:04.975548 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.975377    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8bwm\" (UniqueName: \"kubernetes.io/projected/937fca6a-d8cf-48b9-a0e5-39572043858e-kube-api-access-g8bwm\") pod \"iptables-alerter-tm2rm\" (UID: \"937fca6a-d8cf-48b9-a0e5-39572043858e\") " pod="openshift-network-operator/iptables-alerter-tm2rm"
Sep 18 19:38:04.975548 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.975399    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/db9632d7-9a01-4dc2-a1ac-e1e7505e7b62-var-lib-kubelet\") pod \"tuned-96v5g\" (UID: \"db9632d7-9a01-4dc2-a1ac-e1e7505e7b62\") " pod="openshift-cluster-node-tuning-operator/tuned-96v5g"
Sep 18 19:38:04.975548 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.975407    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f6d3d568-a824-4226-9944-66adc0973032-etc-selinux\") pod \"aws-ebs-csi-driver-node-47hjk\" (UID: \"f6d3d568-a824-4226-9944-66adc0973032\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47hjk"
Sep 18 19:38:04.975548 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.975420    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ecb576b0-eafa-4997-9690-36f2ac243783-agent-certs\") pod \"konnectivity-agent-667km\" (UID: \"ecb576b0-eafa-4997-9690-36f2ac243783\") " pod="kube-system/konnectivity-agent-667km"
Sep 18 19:38:04.975548 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.975448    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db9632d7-9a01-4dc2-a1ac-e1e7505e7b62-etc-kubernetes\") pod \"tuned-96v5g\" (UID: \"db9632d7-9a01-4dc2-a1ac-e1e7505e7b62\") " pod="openshift-cluster-node-tuning-operator/tuned-96v5g"
Sep 18 19:38:04.975548 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.975468    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6df88db0-10f0-429d-b36f-644069df51ed-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-szj4k\" (UID: \"6df88db0-10f0-429d-b36f-644069df51ed\") " pod="openshift-multus/multus-additional-cni-plugins-szj4k"
Sep 18 19:38:04.975548 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.975484    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkpvv\" (UniqueName: \"kubernetes.io/projected/3f7e2771-a1bc-4ac9-9b67-1c7c9473d895-kube-api-access-pkpvv\") pod \"node-resolver-dwwf5\" (UID: \"3f7e2771-a1bc-4ac9-9b67-1c7c9473d895\") " pod="openshift-dns/node-resolver-dwwf5"
Sep 18 19:38:04.975548 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.975501    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/71862b1f-4124-47a1-8942-60acbbb5bdfb-serviceca\") pod \"node-ca-8hdsx\" (UID: \"71862b1f-4124-47a1-8942-60acbbb5bdfb\") " pod="openshift-image-registry/node-ca-8hdsx"
Sep 18 19:38:04.975548 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.975523    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7wk9\" (UniqueName: \"kubernetes.io/projected/71862b1f-4124-47a1-8942-60acbbb5bdfb-kube-api-access-d7wk9\") pod \"node-ca-8hdsx\" (UID: \"71862b1f-4124-47a1-8942-60acbbb5bdfb\") " pod="openshift-image-registry/node-ca-8hdsx"
Sep 18 19:38:04.976095 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.975545    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5jfh\" (UniqueName: \"kubernetes.io/projected/2b852eeb-95f7-40df-adbf-341896362824-kube-api-access-f5jfh\") pod \"multus-vxbw8\" (UID: \"2b852eeb-95f7-40df-adbf-341896362824\") " pod="openshift-multus/multus-vxbw8"
Sep 18 19:38:04.976095 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.975574    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2fd6f85b-2fd5-456a-919c-ca59d7e5203f-host-run-netns\") pod \"ovnkube-node-xcbfz\" (UID: \"2fd6f85b-2fd5-456a-919c-ca59d7e5203f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz"
Sep 18 19:38:04.976095 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.975591    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2fd6f85b-2fd5-456a-919c-ca59d7e5203f-var-lib-openvswitch\") pod \"ovnkube-node-xcbfz\" (UID: \"2fd6f85b-2fd5-456a-919c-ca59d7e5203f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz"
Sep 18 19:38:04.976095 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.975621    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2fd6f85b-2fd5-456a-919c-ca59d7e5203f-etc-openvswitch\") pod \"ovnkube-node-xcbfz\" (UID: \"2fd6f85b-2fd5-456a-919c-ca59d7e5203f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz"
Sep 18 19:38:04.976095 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.975653    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f6d3d568-a824-4226-9944-66adc0973032-registration-dir\") pod \"aws-ebs-csi-driver-node-47hjk\" (UID: \"f6d3d568-a824-4226-9944-66adc0973032\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47hjk"
Sep 18 19:38:04.976095 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.975676    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6df88db0-10f0-429d-b36f-644069df51ed-cni-binary-copy\") pod \"multus-additional-cni-plugins-szj4k\" (UID: \"6df88db0-10f0-429d-b36f-644069df51ed\") " pod="openshift-multus/multus-additional-cni-plugins-szj4k"
Sep 18 19:38:04.976095 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.975698    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f7e2771-a1bc-4ac9-9b67-1c7c9473d895-tmp-dir\") pod \"node-resolver-dwwf5\" (UID: \"3f7e2771-a1bc-4ac9-9b67-1c7c9473d895\") " pod="openshift-dns/node-resolver-dwwf5"
Sep 18 19:38:04.976095 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.975706    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f6d3d568-a824-4226-9944-66adc0973032-registration-dir\") pod \"aws-ebs-csi-driver-node-47hjk\" (UID: \"f6d3d568-a824-4226-9944-66adc0973032\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47hjk"
Sep 18 19:38:04.976095 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.975712    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2b852eeb-95f7-40df-adbf-341896362824-host-run-k8s-cni-cncf-io\") pod \"multus-vxbw8\" (UID: \"2b852eeb-95f7-40df-adbf-341896362824\") " pod="openshift-multus/multus-vxbw8"
Sep 18 19:38:04.976095 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.975744    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b852eeb-95f7-40df-adbf-341896362824-etc-kubernetes\") pod \"multus-vxbw8\" (UID: \"2b852eeb-95f7-40df-adbf-341896362824\") " pod="openshift-multus/multus-vxbw8"
Sep 18 19:38:04.976095 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.975760    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2fd6f85b-2fd5-456a-919c-ca59d7e5203f-node-log\") pod \"ovnkube-node-xcbfz\" (UID: \"2fd6f85b-2fd5-456a-919c-ca59d7e5203f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz"
Sep 18 19:38:04.976095 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.975760    2564 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 18 19:38:04.976095 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.975780    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/db9632d7-9a01-4dc2-a1ac-e1e7505e7b62-etc-modprobe-d\") pod \"tuned-96v5g\" (UID: \"db9632d7-9a01-4dc2-a1ac-e1e7505e7b62\") " pod="openshift-cluster-node-tuning-operator/tuned-96v5g"
Sep 18 19:38:04.976095 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.975798    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/db9632d7-9a01-4dc2-a1ac-e1e7505e7b62-etc-sysconfig\") pod \"tuned-96v5g\" (UID: \"db9632d7-9a01-4dc2-a1ac-e1e7505e7b62\") " pod="openshift-cluster-node-tuning-operator/tuned-96v5g"
Sep 18 19:38:04.976095 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.975817    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/db9632d7-9a01-4dc2-a1ac-e1e7505e7b62-etc-tuned\") pod \"tuned-96v5g\" (UID: \"db9632d7-9a01-4dc2-a1ac-e1e7505e7b62\") " pod="openshift-cluster-node-tuning-operator/tuned-96v5g"
Sep 18 19:38:04.976095 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.975835    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f6d3d568-a824-4226-9944-66adc0973032-kubelet-dir\") pod \"aws-ebs-csi-driver-node-47hjk\" (UID: \"f6d3d568-a824-4226-9944-66adc0973032\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47hjk"
Sep 18 19:38:04.976095 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.975866    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2939ca32-81a7-4e9d-bdc6-ffca73820918-kubelet-config\") pod \"global-pull-secret-syncer-6vdpn\" (UID: \"2939ca32-81a7-4e9d-bdc6-ffca73820918\") " pod="kube-system/global-pull-secret-syncer-6vdpn"
Sep 18 19:38:04.976747 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.975904    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f6d3d568-a824-4226-9944-66adc0973032-kubelet-dir\") pod \"aws-ebs-csi-driver-node-47hjk\" (UID: \"f6d3d568-a824-4226-9944-66adc0973032\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47hjk"
Sep 18 19:38:04.976747 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.975957    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2b852eeb-95f7-40df-adbf-341896362824-host-run-multus-certs\") pod \"multus-vxbw8\" (UID: \"2b852eeb-95f7-40df-adbf-341896362824\") " pod="openshift-multus/multus-vxbw8"
Sep 18 19:38:04.976747 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.976005    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/937fca6a-d8cf-48b9-a0e5-39572043858e-host-slash\") pod \"iptables-alerter-tm2rm\" (UID: \"937fca6a-d8cf-48b9-a0e5-39572043858e\") " pod="openshift-network-operator/iptables-alerter-tm2rm"
Sep 18 19:38:04.976747 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.976025    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2fd6f85b-2fd5-456a-919c-ca59d7e5203f-run-ovn\") pod \"ovnkube-node-xcbfz\" (UID: \"2fd6f85b-2fd5-456a-919c-ca59d7e5203f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz"
Sep 18 19:38:04.976747 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.976071    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2fd6f85b-2fd5-456a-919c-ca59d7e5203f-ovnkube-script-lib\") pod \"ovnkube-node-xcbfz\" (UID: \"2fd6f85b-2fd5-456a-919c-ca59d7e5203f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz"
Sep 18 19:38:04.976747 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.976100    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2b852eeb-95f7-40df-adbf-341896362824-multus-conf-dir\") pod \"multus-vxbw8\" (UID: \"2b852eeb-95f7-40df-adbf-341896362824\") " pod="openshift-multus/multus-vxbw8"
Sep 18 19:38:04.976747 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.976124    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2fd6f85b-2fd5-456a-919c-ca59d7e5203f-run-systemd\") pod \"ovnkube-node-xcbfz\" (UID: \"2fd6f85b-2fd5-456a-919c-ca59d7e5203f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz"
Sep 18 19:38:04.976747 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.976156    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6df88db0-10f0-429d-b36f-644069df51ed-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-szj4k\" (UID: \"6df88db0-10f0-429d-b36f-644069df51ed\") " pod="openshift-multus/multus-additional-cni-plugins-szj4k"
Sep 18 19:38:04.976747 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.976174    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db9632d7-9a01-4dc2-a1ac-e1e7505e7b62-tmp\") pod \"tuned-96v5g\" (UID: \"db9632d7-9a01-4dc2-a1ac-e1e7505e7b62\") " pod="openshift-cluster-node-tuning-operator/tuned-96v5g"
Sep 18 19:38:04.976747 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.976188    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2fd6f85b-2fd5-456a-919c-ca59d7e5203f-run-openvswitch\") pod \"ovnkube-node-xcbfz\" (UID: \"2fd6f85b-2fd5-456a-919c-ca59d7e5203f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz"
Sep 18 19:38:04.976747 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.976201    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2fd6f85b-2fd5-456a-919c-ca59d7e5203f-host-cni-netd\") pod \"ovnkube-node-xcbfz\" (UID: \"2fd6f85b-2fd5-456a-919c-ca59d7e5203f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz"
Sep 18 19:38:04.976747 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.976252    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v67x6\" (UniqueName: \"kubernetes.io/projected/2fd6f85b-2fd5-456a-919c-ca59d7e5203f-kube-api-access-v67x6\") pod \"ovnkube-node-xcbfz\" (UID: \"2fd6f85b-2fd5-456a-919c-ca59d7e5203f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz"
Sep 18 19:38:04.976747 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.976279    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzztj\" (UniqueName: \"kubernetes.io/projected/f6d3d568-a824-4226-9944-66adc0973032-kube-api-access-xzztj\") pod \"aws-ebs-csi-driver-node-47hjk\" (UID: \"f6d3d568-a824-4226-9944-66adc0973032\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47hjk"
Sep 18 19:38:04.976747 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.976317    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/71862b1f-4124-47a1-8942-60acbbb5bdfb-host\") pod \"node-ca-8hdsx\" (UID: \"71862b1f-4124-47a1-8942-60acbbb5bdfb\") " pod="openshift-image-registry/node-ca-8hdsx"
Sep 18 19:38:04.976747 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.976361    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b852eeb-95f7-40df-adbf-341896362824-multus-cni-dir\") pod \"multus-vxbw8\" (UID: \"2b852eeb-95f7-40df-adbf-341896362824\") " pod="openshift-multus/multus-vxbw8"
Sep 18 19:38:04.976747 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.976404    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2b852eeb-95f7-40df-adbf-341896362824-cnibin\") pod \"multus-vxbw8\" (UID: \"2b852eeb-95f7-40df-adbf-341896362824\") " pod="openshift-multus/multus-vxbw8"
Sep 18 19:38:04.976747 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.976433    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2b852eeb-95f7-40df-adbf-341896362824-host-var-lib-cni-bin\") pod \"multus-vxbw8\" (UID: \"2b852eeb-95f7-40df-adbf-341896362824\") " pod="openshift-multus/multus-vxbw8"
Sep 18 19:38:04.977253 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.976463    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b852eeb-95f7-40df-adbf-341896362824-host-var-lib-kubelet\") pod \"multus-vxbw8\" (UID: \"2b852eeb-95f7-40df-adbf-341896362824\") " pod="openshift-multus/multus-vxbw8"
Sep 18 19:38:04.977253 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.976498    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6df88db0-10f0-429d-b36f-644069df51ed-cnibin\") pod \"multus-additional-cni-plugins-szj4k\" (UID: \"6df88db0-10f0-429d-b36f-644069df51ed\") " pod="openshift-multus/multus-additional-cni-plugins-szj4k"
Sep 18 19:38:04.977253 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.976516    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2fd6f85b-2fd5-456a-919c-ca59d7e5203f-log-socket\") pod \"ovnkube-node-xcbfz\" (UID: \"2fd6f85b-2fd5-456a-919c-ca59d7e5203f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz"
Sep 18 19:38:04.977253 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.976530    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/db9632d7-9a01-4dc2-a1ac-e1e7505e7b62-etc-sysctl-d\") pod \"tuned-96v5g\" (UID: \"db9632d7-9a01-4dc2-a1ac-e1e7505e7b62\") " pod="openshift-cluster-node-tuning-operator/tuned-96v5g"
Sep 18 19:38:04.977253 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.976546    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f6d3d568-a824-4226-9944-66adc0973032-sys-fs\") pod \"aws-ebs-csi-driver-node-47hjk\" (UID: \"f6d3d568-a824-4226-9944-66adc0973032\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47hjk"
Sep 18 19:38:04.977253 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.976569    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3f7e2771-a1bc-4ac9-9b67-1c7c9473d895-hosts-file\") pod \"node-resolver-dwwf5\" (UID: \"3f7e2771-a1bc-4ac9-9b67-1c7c9473d895\") " pod="openshift-dns/node-resolver-dwwf5"
Sep 18 19:38:04.977253 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.976597    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2b852eeb-95f7-40df-adbf-341896362824-cni-binary-copy\") pod \"multus-vxbw8\" (UID: \"2b852eeb-95f7-40df-adbf-341896362824\") " pod="openshift-multus/multus-vxbw8"
Sep 18 19:38:04.977253 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.976619    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2b852eeb-95f7-40df-adbf-341896362824-multus-socket-dir-parent\") pod \"multus-vxbw8\" (UID: \"2b852eeb-95f7-40df-adbf-341896362824\") " pod="openshift-multus/multus-vxbw8"
Sep 18 19:38:04.977253 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.976619    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f6d3d568-a824-4226-9944-66adc0973032-sys-fs\") pod \"aws-ebs-csi-driver-node-47hjk\" (UID: \"f6d3d568-a824-4226-9944-66adc0973032\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47hjk"
Sep 18 19:38:04.979591 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.979568    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ecb576b0-eafa-4997-9690-36f2ac243783-agent-certs\") pod \"konnectivity-agent-667km\" (UID: \"ecb576b0-eafa-4997-9690-36f2ac243783\") " pod="kube-system/konnectivity-agent-667km"
Sep 18 19:38:04.983440 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.983418    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzztj\" (UniqueName: \"kubernetes.io/projected/f6d3d568-a824-4226-9944-66adc0973032-kube-api-access-xzztj\") pod \"aws-ebs-csi-driver-node-47hjk\" (UID: \"f6d3d568-a824-4226-9944-66adc0973032\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47hjk"
Sep 18 19:38:04.983516 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.983495    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwfbj\" (UniqueName: \"kubernetes.io/projected/256a72d8-e079-42fb-9426-c53adc283d1f-kube-api-access-wwfbj\") pod \"network-metrics-daemon-xgxbn\" (UID: \"256a72d8-e079-42fb-9426-c53adc283d1f\") " pod="openshift-multus/network-metrics-daemon-xgxbn"
Sep 18 19:38:04.997532 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.997489    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-161.ec2.internal" event={"ID":"9eb8c7aafdc391f2f7ae0f04a98530ea","Type":"ContainerStarted","Data":"ba8af836b80c3721c9fdd45b677e19350b041c34e11be86736e86d3bc773cffd"}
Sep 18 19:38:04.998575 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:04.998542    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.ec2.internal" event={"ID":"2c1e9c7cacf21e14413fce157c96f3cd","Type":"ContainerStarted","Data":"0334d70254f30993173e3d99e3accfb83aa287555694934633ebeec541a62a49"}
Sep 18 19:38:05.077808 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.077775    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2b852eeb-95f7-40df-adbf-341896362824-host-run-multus-certs\") pod \"multus-vxbw8\" (UID: \"2b852eeb-95f7-40df-adbf-341896362824\") " pod="openshift-multus/multus-vxbw8"
Sep 18 19:38:05.077808 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.077809    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/937fca6a-d8cf-48b9-a0e5-39572043858e-host-slash\") pod \"iptables-alerter-tm2rm\" (UID: \"937fca6a-d8cf-48b9-a0e5-39572043858e\") " pod="openshift-network-operator/iptables-alerter-tm2rm"
Sep 18 19:38:05.078092 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.077825    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2fd6f85b-2fd5-456a-919c-ca59d7e5203f-run-ovn\") pod \"ovnkube-node-xcbfz\" (UID: \"2fd6f85b-2fd5-456a-919c-ca59d7e5203f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz"
Sep 18 19:38:05.078092 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.077842    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2fd6f85b-2fd5-456a-919c-ca59d7e5203f-ovnkube-script-lib\") pod \"ovnkube-node-xcbfz\" (UID: \"2fd6f85b-2fd5-456a-919c-ca59d7e5203f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz"
Sep 18 19:38:05.078092 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.077856    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2b852eeb-95f7-40df-adbf-341896362824-multus-conf-dir\") pod \"multus-vxbw8\" (UID: \"2b852eeb-95f7-40df-adbf-341896362824\") " pod="openshift-multus/multus-vxbw8"
Sep 18 19:38:05.078092 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.077873    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2fd6f85b-2fd5-456a-919c-ca59d7e5203f-run-systemd\") pod \"ovnkube-node-xcbfz\" (UID: \"2fd6f85b-2fd5-456a-919c-ca59d7e5203f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz"
Sep 18 19:38:05.078092 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.077889    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6df88db0-10f0-429d-b36f-644069df51ed-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-szj4k\" (UID: \"6df88db0-10f0-429d-b36f-644069df51ed\") " pod="openshift-multus/multus-additional-cni-plugins-szj4k"
Sep 18 19:38:05.078092 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.077900    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2b852eeb-95f7-40df-adbf-341896362824-host-run-multus-certs\") pod \"multus-vxbw8\" (UID: \"2b852eeb-95f7-40df-adbf-341896362824\") " pod="openshift-multus/multus-vxbw8"
Sep 18 19:38:05.078092 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.077926    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2b852eeb-95f7-40df-adbf-341896362824-multus-conf-dir\") pod \"multus-vxbw8\" (UID: \"2b852eeb-95f7-40df-adbf-341896362824\") " pod="openshift-multus/multus-vxbw8"
Sep 18 19:38:05.078092 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.077941    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2fd6f85b-2fd5-456a-919c-ca59d7e5203f-run-systemd\") pod \"ovnkube-node-xcbfz\" (UID: \"2fd6f85b-2fd5-456a-919c-ca59d7e5203f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz"
Sep 18 19:38:05.078092 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.077926    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2fd6f85b-2fd5-456a-919c-ca59d7e5203f-run-ovn\") pod \"ovnkube-node-xcbfz\" (UID: \"2fd6f85b-2fd5-456a-919c-ca59d7e5203f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz"
Sep 18 19:38:05.078092 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.077899    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/937fca6a-d8cf-48b9-a0e5-39572043858e-host-slash\") pod \"iptables-alerter-tm2rm\" (UID: \"937fca6a-d8cf-48b9-a0e5-39572043858e\") " pod="openshift-network-operator/iptables-alerter-tm2rm"
Sep 18 19:38:05.078092 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.077912    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db9632d7-9a01-4dc2-a1ac-e1e7505e7b62-tmp\") pod \"tuned-96v5g\" (UID: \"db9632d7-9a01-4dc2-a1ac-e1e7505e7b62\") " pod="openshift-cluster-node-tuning-operator/tuned-96v5g"
Sep 18 19:38:05.078092 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.078028    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2fd6f85b-2fd5-456a-919c-ca59d7e5203f-run-openvswitch\") pod \"ovnkube-node-xcbfz\" (UID: \"2fd6f85b-2fd5-456a-919c-ca59d7e5203f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz"
Sep 18 19:38:05.078092 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.078060    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2fd6f85b-2fd5-456a-919c-ca59d7e5203f-host-cni-netd\") pod \"ovnkube-node-xcbfz\" (UID: \"2fd6f85b-2fd5-456a-919c-ca59d7e5203f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz"
Sep 18 19:38:05.078092 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.078085    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v67x6\" (UniqueName: \"kubernetes.io/projected/2fd6f85b-2fd5-456a-919c-ca59d7e5203f-kube-api-access-v67x6\") pod \"ovnkube-node-xcbfz\" (UID: \"2fd6f85b-2fd5-456a-919c-ca59d7e5203f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz"
Sep 18 19:38:05.078732 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.078110    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/71862b1f-4124-47a1-8942-60acbbb5bdfb-host\") pod \"node-ca-8hdsx\" (UID: \"71862b1f-4124-47a1-8942-60acbbb5bdfb\") " pod="openshift-image-registry/node-ca-8hdsx"
Sep 18 19:38:05.078732 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.078059    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2fd6f85b-2fd5-456a-919c-ca59d7e5203f-run-openvswitch\") pod \"ovnkube-node-xcbfz\" (UID: \"2fd6f85b-2fd5-456a-919c-ca59d7e5203f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz"
Sep 18 19:38:05.078732 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.078132    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b852eeb-95f7-40df-adbf-341896362824-multus-cni-dir\") pod \"multus-vxbw8\" (UID: \"2b852eeb-95f7-40df-adbf-341896362824\") " pod="openshift-multus/multus-vxbw8"
Sep 18 19:38:05.078732 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.078156    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2b852eeb-95f7-40df-adbf-341896362824-cnibin\") pod \"multus-vxbw8\" (UID: \"2b852eeb-95f7-40df-adbf-341896362824\") " pod="openshift-multus/multus-vxbw8"
Sep 18 19:38:05.078732 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.078166    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2fd6f85b-2fd5-456a-919c-ca59d7e5203f-host-cni-netd\") pod \"ovnkube-node-xcbfz\" (UID: \"2fd6f85b-2fd5-456a-919c-ca59d7e5203f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz"
Sep 18 19:38:05.078732 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.078180    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2b852eeb-95f7-40df-adbf-341896362824-host-var-lib-cni-bin\") pod \"multus-vxbw8\" (UID: \"2b852eeb-95f7-40df-adbf-341896362824\") " pod="openshift-multus/multus-vxbw8"
Sep 18 19:38:05.078732 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.078198    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/71862b1f-4124-47a1-8942-60acbbb5bdfb-host\") pod \"node-ca-8hdsx\" (UID: \"71862b1f-4124-47a1-8942-60acbbb5bdfb\") " pod="openshift-image-registry/node-ca-8hdsx"
Sep 18 19:38:05.078732 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.078228    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b852eeb-95f7-40df-adbf-341896362824-multus-cni-dir\") pod \"multus-vxbw8\" (UID: \"2b852eeb-95f7-40df-adbf-341896362824\") " pod="openshift-multus/multus-vxbw8"
Sep 18 19:38:05.078732 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.078205    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b852eeb-95f7-40df-adbf-341896362824-host-var-lib-kubelet\") pod \"multus-vxbw8\" (UID: \"2b852eeb-95f7-40df-adbf-341896362824\") " pod="openshift-multus/multus-vxbw8"
Sep 18 19:38:05.078732 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.078246    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b852eeb-95f7-40df-adbf-341896362824-host-var-lib-kubelet\") pod \"multus-vxbw8\" (UID: \"2b852eeb-95f7-40df-adbf-341896362824\") " pod="openshift-multus/multus-vxbw8"
Sep 18 19:38:05.078732 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.078274    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2b852eeb-95f7-40df-adbf-341896362824-cnibin\") pod \"multus-vxbw8\" (UID: \"2b852eeb-95f7-40df-adbf-341896362824\") " pod="openshift-multus/multus-vxbw8"
Sep 18 19:38:05.078732 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.078273    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6df88db0-10f0-429d-b36f-644069df51ed-cnibin\") pod \"multus-additional-cni-plugins-szj4k\" (UID: \"6df88db0-10f0-429d-b36f-644069df51ed\") " pod="openshift-multus/multus-additional-cni-plugins-szj4k"
Sep 18 19:38:05.078732 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.078291    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2b852eeb-95f7-40df-adbf-341896362824-host-var-lib-cni-bin\") pod \"multus-vxbw8\" (UID: \"2b852eeb-95f7-40df-adbf-341896362824\") " pod="openshift-multus/multus-vxbw8"
Sep 18 19:38:05.078732 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.078310    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2fd6f85b-2fd5-456a-919c-ca59d7e5203f-log-socket\") pod \"ovnkube-node-xcbfz\" (UID: \"2fd6f85b-2fd5-456a-919c-ca59d7e5203f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz"
Sep 18 19:38:05.078732 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.078324    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6df88db0-10f0-429d-b36f-644069df51ed-cnibin\") pod \"multus-additional-cni-plugins-szj4k\" (UID: \"6df88db0-10f0-429d-b36f-644069df51ed\") " pod="openshift-multus/multus-additional-cni-plugins-szj4k"
Sep 18 19:38:05.078732 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.078334    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/db9632d7-9a01-4dc2-a1ac-e1e7505e7b62-etc-sysctl-d\") pod \"tuned-96v5g\" (UID: \"db9632d7-9a01-4dc2-a1ac-e1e7505e7b62\") " pod="openshift-cluster-node-tuning-operator/tuned-96v5g"
Sep 18 19:38:05.078732 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.078357    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3f7e2771-a1bc-4ac9-9b67-1c7c9473d895-hosts-file\") pod \"node-resolver-dwwf5\" (UID: \"3f7e2771-a1bc-4ac9-9b67-1c7c9473d895\") " pod="openshift-dns/node-resolver-dwwf5"
Sep 18 19:38:05.078732 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.078360    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2fd6f85b-2fd5-456a-919c-ca59d7e5203f-log-socket\") pod \"ovnkube-node-xcbfz\" (UID: \"2fd6f85b-2fd5-456a-919c-ca59d7e5203f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz"
Sep 18 19:38:05.079628 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.078378    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2b852eeb-95f7-40df-adbf-341896362824-cni-binary-copy\") pod \"multus-vxbw8\" (UID: \"2b852eeb-95f7-40df-adbf-341896362824\") " pod="openshift-multus/multus-vxbw8"
Sep 18 19:38:05.079628 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.078406    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2b852eeb-95f7-40df-adbf-341896362824-multus-socket-dir-parent\") pod \"multus-vxbw8\" (UID: \"2b852eeb-95f7-40df-adbf-341896362824\") " pod="openshift-multus/multus-vxbw8"
Sep 18 19:38:05.079628 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.078426    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/db9632d7-9a01-4dc2-a1ac-e1e7505e7b62-lib-modules\") pod \"tuned-96v5g\" (UID: \"db9632d7-9a01-4dc2-a1ac-e1e7505e7b62\") " pod="openshift-cluster-node-tuning-operator/tuned-96v5g"
Sep 18 19:38:05.079628 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.078443    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjgc5\" (UniqueName: \"kubernetes.io/projected/1b42bdad-4122-4a0a-978c-d2d5a7bdbd55-kube-api-access-zjgc5\") pod \"network-check-target-857bd\" (UID: \"1b42bdad-4122-4a0a-978c-d2d5a7bdbd55\") " pod="openshift-network-diagnostics/network-check-target-857bd"
Sep 18 19:38:05.079628 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.078450    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3f7e2771-a1bc-4ac9-9b67-1c7c9473d895-hosts-file\") pod \"node-resolver-dwwf5\" (UID: \"3f7e2771-a1bc-4ac9-9b67-1c7c9473d895\") " pod="openshift-dns/node-resolver-dwwf5"
Sep 18 19:38:05.079628 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.078467    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2fd6f85b-2fd5-456a-919c-ca59d7e5203f-systemd-units\") pod \"ovnkube-node-xcbfz\" (UID: \"2fd6f85b-2fd5-456a-919c-ca59d7e5203f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz"
Sep 18 19:38:05.079628 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.078501    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2b852eeb-95f7-40df-adbf-341896362824-host-run-netns\") pod \"multus-vxbw8\" (UID: \"2b852eeb-95f7-40df-adbf-341896362824\") " pod="openshift-multus/multus-vxbw8"
Sep 18 19:38:05.079628 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.078509    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/db9632d7-9a01-4dc2-a1ac-e1e7505e7b62-etc-sysctl-d\") pod \"tuned-96v5g\" (UID: \"db9632d7-9a01-4dc2-a1ac-e1e7505e7b62\") " pod="openshift-cluster-node-tuning-operator/tuned-96v5g"
Sep 18 19:38:05.079628 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.078522    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2b852eeb-95f7-40df-adbf-341896362824-hostroot\") pod \"multus-vxbw8\" (UID: \"2b852eeb-95f7-40df-adbf-341896362824\") " pod="openshift-multus/multus-vxbw8"
Sep 18 19:38:05.079628 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.078525    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6df88db0-10f0-429d-b36f-644069df51ed-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-szj4k\" (UID: \"6df88db0-10f0-429d-b36f-644069df51ed\") " pod="openshift-multus/multus-additional-cni-plugins-szj4k"
Sep 18 19:38:05.079628 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.078535    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2fd6f85b-2fd5-456a-919c-ca59d7e5203f-systemd-units\") pod \"ovnkube-node-xcbfz\" (UID: \"2fd6f85b-2fd5-456a-919c-ca59d7e5203f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz"
Sep 18 19:38:05.079628 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.078544    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2fd6f85b-2fd5-456a-919c-ca59d7e5203f-host-slash\") pod \"ovnkube-node-xcbfz\" (UID: \"2fd6f85b-2fd5-456a-919c-ca59d7e5203f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz"
Sep 18 19:38:05.079628 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.078500    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2b852eeb-95f7-40df-adbf-341896362824-multus-socket-dir-parent\") pod \"multus-vxbw8\" (UID: \"2b852eeb-95f7-40df-adbf-341896362824\") " pod="openshift-multus/multus-vxbw8"
Sep 18 19:38:05.079628 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.078560    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2b852eeb-95f7-40df-adbf-341896362824-host-run-netns\") pod \"multus-vxbw8\" (UID: \"2b852eeb-95f7-40df-adbf-341896362824\") " pod="openshift-multus/multus-vxbw8"
Sep 18 19:38:05.079628 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.078575    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2fd6f85b-2fd5-456a-919c-ca59d7e5203f-host-cni-bin\") pod \"ovnkube-node-xcbfz\" (UID: \"2fd6f85b-2fd5-456a-919c-ca59d7e5203f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz"
Sep 18 19:38:05.079628 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.078580    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/db9632d7-9a01-4dc2-a1ac-e1e7505e7b62-lib-modules\") pod \"tuned-96v5g\" (UID: \"db9632d7-9a01-4dc2-a1ac-e1e7505e7b62\") " pod="openshift-cluster-node-tuning-operator/tuned-96v5g"
Sep 18 19:38:05.079628 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.078600    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2b852eeb-95f7-40df-adbf-341896362824-hostroot\") pod \"multus-vxbw8\" (UID: \"2b852eeb-95f7-40df-adbf-341896362824\") " pod="openshift-multus/multus-vxbw8"
Sep 18 19:38:05.079628 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.078603    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2fd6f85b-2fd5-456a-919c-ca59d7e5203f-host-slash\") pod \"ovnkube-node-xcbfz\" (UID: \"2fd6f85b-2fd5-456a-919c-ca59d7e5203f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz"
Sep 18 19:38:05.080486 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.078618    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2fd6f85b-2fd5-456a-919c-ca59d7e5203f-host-cni-bin\") pod \"ovnkube-node-xcbfz\" (UID: \"2fd6f85b-2fd5-456a-919c-ca59d7e5203f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz"
Sep 18 19:38:05.080486 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.078631    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2fd6f85b-2fd5-456a-919c-ca59d7e5203f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xcbfz\" (UID: \"2fd6f85b-2fd5-456a-919c-ca59d7e5203f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz"
Sep 18 19:38:05.080486 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.078656    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2fd6f85b-2fd5-456a-919c-ca59d7e5203f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xcbfz\" (UID: \"2fd6f85b-2fd5-456a-919c-ca59d7e5203f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz"
Sep 18 19:38:05.080486 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.078663    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2fd6f85b-2fd5-456a-919c-ca59d7e5203f-ovn-node-metrics-cert\") pod \"ovnkube-node-xcbfz\" (UID: \"2fd6f85b-2fd5-456a-919c-ca59d7e5203f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz"
Sep 18 19:38:05.080486 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.078703    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2939ca32-81a7-4e9d-bdc6-ffca73820918-original-pull-secret\") pod \"global-pull-secret-syncer-6vdpn\" (UID: \"2939ca32-81a7-4e9d-bdc6-ffca73820918\") " pod="kube-system/global-pull-secret-syncer-6vdpn"
Sep 18 19:38:05.080486 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.078720    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6df88db0-10f0-429d-b36f-644069df51ed-system-cni-dir\") pod \"multus-additional-cni-plugins-szj4k\" (UID: \"6df88db0-10f0-429d-b36f-644069df51ed\") " pod="openshift-multus/multus-additional-cni-plugins-szj4k"
Sep 18 19:38:05.080486 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.078742    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5m8v\" (UniqueName: \"kubernetes.io/projected/6df88db0-10f0-429d-b36f-644069df51ed-kube-api-access-p5m8v\") pod \"multus-additional-cni-plugins-szj4k\" (UID: \"6df88db0-10f0-429d-b36f-644069df51ed\") " pod="openshift-multus/multus-additional-cni-plugins-szj4k"
Sep 18 19:38:05.080486 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.078769    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/937fca6a-d8cf-48b9-a0e5-39572043858e-iptables-alerter-script\") pod \"iptables-alerter-tm2rm\" (UID: \"937fca6a-d8cf-48b9-a0e5-39572043858e\") " pod="openshift-network-operator/iptables-alerter-tm2rm"
Sep 18 19:38:05.080486 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.078794    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/db9632d7-9a01-4dc2-a1ac-e1e7505e7b62-etc-sysctl-conf\") pod \"tuned-96v5g\" (UID: \"db9632d7-9a01-4dc2-a1ac-e1e7505e7b62\") " pod="openshift-cluster-node-tuning-operator/tuned-96v5g"
Sep 18 19:38:05.080486 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.078803    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6df88db0-10f0-429d-b36f-644069df51ed-system-cni-dir\") pod \"multus-additional-cni-plugins-szj4k\" (UID: \"6df88db0-10f0-429d-b36f-644069df51ed\") " pod="openshift-multus/multus-additional-cni-plugins-szj4k"
Sep 18 19:38:05.080486 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:05.078808    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:05.080486 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:05.078921    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2939ca32-81a7-4e9d-bdc6-ffca73820918-original-pull-secret podName:2939ca32-81a7-4e9d-bdc6-ffca73820918 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:05.578902355 +0000 UTC m=+2.089258438 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2939ca32-81a7-4e9d-bdc6-ffca73820918-original-pull-secret") pod "global-pull-secret-syncer-6vdpn" (UID: "2939ca32-81a7-4e9d-bdc6-ffca73820918") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:05.080486 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.078921    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/db9632d7-9a01-4dc2-a1ac-e1e7505e7b62-etc-sysctl-conf\") pod \"tuned-96v5g\" (UID: \"db9632d7-9a01-4dc2-a1ac-e1e7505e7b62\") " pod="openshift-cluster-node-tuning-operator/tuned-96v5g"
Sep 18 19:38:05.080486 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.079001    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2b852eeb-95f7-40df-adbf-341896362824-cni-binary-copy\") pod \"multus-vxbw8\" (UID: \"2b852eeb-95f7-40df-adbf-341896362824\") " pod="openshift-multus/multus-vxbw8"
Sep 18 19:38:05.080486 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.079019    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/db9632d7-9a01-4dc2-a1ac-e1e7505e7b62-run\") pod \"tuned-96v5g\" (UID: \"db9632d7-9a01-4dc2-a1ac-e1e7505e7b62\") " pod="openshift-cluster-node-tuning-operator/tuned-96v5g"
Sep 18 19:38:05.080486 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.079066    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/db9632d7-9a01-4dc2-a1ac-e1e7505e7b62-run\") pod \"tuned-96v5g\" (UID: \"db9632d7-9a01-4dc2-a1ac-e1e7505e7b62\") " pod="openshift-cluster-node-tuning-operator/tuned-96v5g"
Sep 18 19:38:05.080486 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.079086    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2939ca32-81a7-4e9d-bdc6-ffca73820918-dbus\") pod \"global-pull-secret-syncer-6vdpn\" (UID: \"2939ca32-81a7-4e9d-bdc6-ffca73820918\") " pod="kube-system/global-pull-secret-syncer-6vdpn"
Sep 18 19:38:05.081189 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.079112    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2b852eeb-95f7-40df-adbf-341896362824-host-var-lib-cni-multus\") pod \"multus-vxbw8\" (UID: \"2b852eeb-95f7-40df-adbf-341896362824\") " pod="openshift-multus/multus-vxbw8"
Sep 18 19:38:05.081189 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.079136    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2b852eeb-95f7-40df-adbf-341896362824-multus-daemon-config\") pod \"multus-vxbw8\" (UID: \"2b852eeb-95f7-40df-adbf-341896362824\") " pod="openshift-multus/multus-vxbw8"
Sep 18 19:38:05.081189 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.079159    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/db9632d7-9a01-4dc2-a1ac-e1e7505e7b62-sys\") pod \"tuned-96v5g\" (UID: \"db9632d7-9a01-4dc2-a1ac-e1e7505e7b62\") " pod="openshift-cluster-node-tuning-operator/tuned-96v5g"
Sep 18 19:38:05.081189 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.079161    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2939ca32-81a7-4e9d-bdc6-ffca73820918-dbus\") pod \"global-pull-secret-syncer-6vdpn\" (UID: \"2939ca32-81a7-4e9d-bdc6-ffca73820918\") " pod="kube-system/global-pull-secret-syncer-6vdpn"
Sep 18 19:38:05.081189 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.079199    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/db9632d7-9a01-4dc2-a1ac-e1e7505e7b62-host\") pod \"tuned-96v5g\" (UID: \"db9632d7-9a01-4dc2-a1ac-e1e7505e7b62\") " pod="openshift-cluster-node-tuning-operator/tuned-96v5g"
Sep 18 19:38:05.081189 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.079202    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2b852eeb-95f7-40df-adbf-341896362824-host-var-lib-cni-multus\") pod \"multus-vxbw8\" (UID: \"2b852eeb-95f7-40df-adbf-341896362824\") " pod="openshift-multus/multus-vxbw8"
Sep 18 19:38:05.081189 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.079225    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2fd6f85b-2fd5-456a-919c-ca59d7e5203f-host-kubelet\") pod \"ovnkube-node-xcbfz\" (UID: \"2fd6f85b-2fd5-456a-919c-ca59d7e5203f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz"
Sep 18 19:38:05.081189 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.079253    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b852eeb-95f7-40df-adbf-341896362824-system-cni-dir\") pod \"multus-vxbw8\" (UID: \"2b852eeb-95f7-40df-adbf-341896362824\") " pod="openshift-multus/multus-vxbw8"
Sep 18 19:38:05.081189 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.079276    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2fd6f85b-2fd5-456a-919c-ca59d7e5203f-ovnkube-config\") pod \"ovnkube-node-xcbfz\" (UID: \"2fd6f85b-2fd5-456a-919c-ca59d7e5203f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz"
Sep 18 19:38:05.081189 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.079291    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/db9632d7-9a01-4dc2-a1ac-e1e7505e7b62-host\") pod \"tuned-96v5g\" (UID: \"db9632d7-9a01-4dc2-a1ac-e1e7505e7b62\") " pod="openshift-cluster-node-tuning-operator/tuned-96v5g"
Sep 18 19:38:05.081189 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.079308    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2fd6f85b-2fd5-456a-919c-ca59d7e5203f-ovnkube-script-lib\") pod \"ovnkube-node-xcbfz\" (UID: \"2fd6f85b-2fd5-456a-919c-ca59d7e5203f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz"
Sep 18 19:38:05.081189 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.079309    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/db9632d7-9a01-4dc2-a1ac-e1e7505e7b62-sys\") pod \"tuned-96v5g\" (UID: \"db9632d7-9a01-4dc2-a1ac-e1e7505e7b62\") " pod="openshift-cluster-node-tuning-operator/tuned-96v5g"
Sep 18 19:38:05.081189 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.079313    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/db9632d7-9a01-4dc2-a1ac-e1e7505e7b62-etc-systemd\") pod \"tuned-96v5g\" (UID: \"db9632d7-9a01-4dc2-a1ac-e1e7505e7b62\") " pod="openshift-cluster-node-tuning-operator/tuned-96v5g"
Sep 18 19:38:05.081189 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.079353    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/db9632d7-9a01-4dc2-a1ac-e1e7505e7b62-etc-systemd\") pod \"tuned-96v5g\" (UID: \"db9632d7-9a01-4dc2-a1ac-e1e7505e7b62\") " pod="openshift-cluster-node-tuning-operator/tuned-96v5g"
Sep 18 19:38:05.081189 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.079373    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q2lc\" (UniqueName: \"kubernetes.io/projected/db9632d7-9a01-4dc2-a1ac-e1e7505e7b62-kube-api-access-2q2lc\") pod \"tuned-96v5g\" (UID: \"db9632d7-9a01-4dc2-a1ac-e1e7505e7b62\") " pod="openshift-cluster-node-tuning-operator/tuned-96v5g"
Sep 18 19:38:05.081189 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.079391    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/937fca6a-d8cf-48b9-a0e5-39572043858e-iptables-alerter-script\") pod \"iptables-alerter-tm2rm\" (UID: \"937fca6a-d8cf-48b9-a0e5-39572043858e\") " pod="openshift-network-operator/iptables-alerter-tm2rm"
Sep 18 19:38:05.081189 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.079400    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2fd6f85b-2fd5-456a-919c-ca59d7e5203f-host-run-ovn-kubernetes\") pod \"ovnkube-node-xcbfz\" (UID: \"2fd6f85b-2fd5-456a-919c-ca59d7e5203f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz"
Sep 18 19:38:05.081189 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.079419    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b852eeb-95f7-40df-adbf-341896362824-system-cni-dir\") pod \"multus-vxbw8\" (UID: \"2b852eeb-95f7-40df-adbf-341896362824\") " pod="openshift-multus/multus-vxbw8"
Sep 18 19:38:05.081687 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.079433    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2fd6f85b-2fd5-456a-919c-ca59d7e5203f-env-overrides\") pod \"ovnkube-node-xcbfz\" (UID: \"2fd6f85b-2fd5-456a-919c-ca59d7e5203f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz"
Sep 18 19:38:05.081687 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.079462    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6df88db0-10f0-429d-b36f-644069df51ed-tuning-conf-dir\") pod \"multus-additional-cni-plugins-szj4k\" (UID: \"6df88db0-10f0-429d-b36f-644069df51ed\") " pod="openshift-multus/multus-additional-cni-plugins-szj4k"
Sep 18 19:38:05.081687 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.079436    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2fd6f85b-2fd5-456a-919c-ca59d7e5203f-host-run-ovn-kubernetes\") pod \"ovnkube-node-xcbfz\" (UID: \"2fd6f85b-2fd5-456a-919c-ca59d7e5203f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz"
Sep 18 19:38:05.081687 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.079487    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8bwm\" (UniqueName: \"kubernetes.io/projected/937fca6a-d8cf-48b9-a0e5-39572043858e-kube-api-access-g8bwm\") pod \"iptables-alerter-tm2rm\" (UID: \"937fca6a-d8cf-48b9-a0e5-39572043858e\") " pod="openshift-network-operator/iptables-alerter-tm2rm"
Sep 18 19:38:05.081687 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.079510    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/db9632d7-9a01-4dc2-a1ac-e1e7505e7b62-var-lib-kubelet\") pod \"tuned-96v5g\" (UID: \"db9632d7-9a01-4dc2-a1ac-e1e7505e7b62\") " pod="openshift-cluster-node-tuning-operator/tuned-96v5g"
Sep 18 19:38:05.081687 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.079536    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db9632d7-9a01-4dc2-a1ac-e1e7505e7b62-etc-kubernetes\") pod \"tuned-96v5g\" (UID: \"db9632d7-9a01-4dc2-a1ac-e1e7505e7b62\") " pod="openshift-cluster-node-tuning-operator/tuned-96v5g"
Sep 18 19:38:05.081687 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.079568    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6df88db0-10f0-429d-b36f-644069df51ed-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-szj4k\" (UID: \"6df88db0-10f0-429d-b36f-644069df51ed\") " pod="openshift-multus/multus-additional-cni-plugins-szj4k"
Sep 18 19:38:05.081687 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.079595    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkpvv\" (UniqueName: \"kubernetes.io/projected/3f7e2771-a1bc-4ac9-9b67-1c7c9473d895-kube-api-access-pkpvv\") pod \"node-resolver-dwwf5\" (UID: \"3f7e2771-a1bc-4ac9-9b67-1c7c9473d895\") " pod="openshift-dns/node-resolver-dwwf5"
Sep 18 19:38:05.081687 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.079618    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/71862b1f-4124-47a1-8942-60acbbb5bdfb-serviceca\") pod \"node-ca-8hdsx\" (UID: \"71862b1f-4124-47a1-8942-60acbbb5bdfb\") " pod="openshift-image-registry/node-ca-8hdsx"
Sep 18 19:38:05.081687 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.079645    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7wk9\" (UniqueName: \"kubernetes.io/projected/71862b1f-4124-47a1-8942-60acbbb5bdfb-kube-api-access-d7wk9\") pod \"node-ca-8hdsx\" (UID: \"71862b1f-4124-47a1-8942-60acbbb5bdfb\") " pod="openshift-image-registry/node-ca-8hdsx"
Sep 18 19:38:05.081687 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.079657    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/db9632d7-9a01-4dc2-a1ac-e1e7505e7b62-var-lib-kubelet\") pod \"tuned-96v5g\" (UID: \"db9632d7-9a01-4dc2-a1ac-e1e7505e7b62\") " pod="openshift-cluster-node-tuning-operator/tuned-96v5g"
Sep 18 19:38:05.081687 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.079673    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5jfh\" (UniqueName: \"kubernetes.io/projected/2b852eeb-95f7-40df-adbf-341896362824-kube-api-access-f5jfh\") pod \"multus-vxbw8\" (UID: \"2b852eeb-95f7-40df-adbf-341896362824\") " pod="openshift-multus/multus-vxbw8"
Sep 18 19:38:05.081687 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.079704    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2fd6f85b-2fd5-456a-919c-ca59d7e5203f-host-run-netns\") pod \"ovnkube-node-xcbfz\" (UID: \"2fd6f85b-2fd5-456a-919c-ca59d7e5203f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz"
Sep 18 19:38:05.081687 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.079726    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2fd6f85b-2fd5-456a-919c-ca59d7e5203f-var-lib-openvswitch\") pod \"ovnkube-node-xcbfz\" (UID: \"2fd6f85b-2fd5-456a-919c-ca59d7e5203f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz"
Sep 18 19:38:05.081687 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.079747    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2fd6f85b-2fd5-456a-919c-ca59d7e5203f-etc-openvswitch\") pod \"ovnkube-node-xcbfz\" (UID: \"2fd6f85b-2fd5-456a-919c-ca59d7e5203f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz"
Sep 18 19:38:05.081687 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.079771    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6df88db0-10f0-429d-b36f-644069df51ed-cni-binary-copy\") pod \"multus-additional-cni-plugins-szj4k\" (UID: \"6df88db0-10f0-429d-b36f-644069df51ed\") " pod="openshift-multus/multus-additional-cni-plugins-szj4k"
Sep 18 19:38:05.081687 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.079816    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f7e2771-a1bc-4ac9-9b67-1c7c9473d895-tmp-dir\") pod \"node-resolver-dwwf5\" (UID: \"3f7e2771-a1bc-4ac9-9b67-1c7c9473d895\") " pod="openshift-dns/node-resolver-dwwf5"
Sep 18 19:38:05.082226 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.079840    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2b852eeb-95f7-40df-adbf-341896362824-host-run-k8s-cni-cncf-io\") pod \"multus-vxbw8\" (UID: \"2b852eeb-95f7-40df-adbf-341896362824\") " pod="openshift-multus/multus-vxbw8"
Sep 18 19:38:05.082226 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.079829    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6df88db0-10f0-429d-b36f-644069df51ed-tuning-conf-dir\") pod \"multus-additional-cni-plugins-szj4k\" (UID: \"6df88db0-10f0-429d-b36f-644069df51ed\") " pod="openshift-multus/multus-additional-cni-plugins-szj4k"
Sep 18 19:38:05.082226 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.079463    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2fd6f85b-2fd5-456a-919c-ca59d7e5203f-host-kubelet\") pod \"ovnkube-node-xcbfz\" (UID: \"2fd6f85b-2fd5-456a-919c-ca59d7e5203f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz"
Sep 18 19:38:05.082226 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.079896    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2b852eeb-95f7-40df-adbf-341896362824-host-run-k8s-cni-cncf-io\") pod \"multus-vxbw8\" (UID: \"2b852eeb-95f7-40df-adbf-341896362824\") " pod="openshift-multus/multus-vxbw8"
Sep 18 19:38:05.082226 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.079900    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2fd6f85b-2fd5-456a-919c-ca59d7e5203f-var-lib-openvswitch\") pod \"ovnkube-node-xcbfz\" (UID: \"2fd6f85b-2fd5-456a-919c-ca59d7e5203f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz"
Sep 18 19:38:05.082226 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.079913    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2fd6f85b-2fd5-456a-919c-ca59d7e5203f-ovnkube-config\") pod \"ovnkube-node-xcbfz\" (UID: \"2fd6f85b-2fd5-456a-919c-ca59d7e5203f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz"
Sep 18 19:38:05.082226 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.079947    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2fd6f85b-2fd5-456a-919c-ca59d7e5203f-etc-openvswitch\") pod \"ovnkube-node-xcbfz\" (UID: \"2fd6f85b-2fd5-456a-919c-ca59d7e5203f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz"
Sep 18 19:38:05.082226 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.079963    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db9632d7-9a01-4dc2-a1ac-e1e7505e7b62-etc-kubernetes\") pod \"tuned-96v5g\" (UID: \"db9632d7-9a01-4dc2-a1ac-e1e7505e7b62\") " pod="openshift-cluster-node-tuning-operator/tuned-96v5g"
Sep 18 19:38:05.082226 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.080157    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2fd6f85b-2fd5-456a-919c-ca59d7e5203f-host-run-netns\") pod \"ovnkube-node-xcbfz\" (UID: \"2fd6f85b-2fd5-456a-919c-ca59d7e5203f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz"
Sep 18 19:38:05.082226 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.080184    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6df88db0-10f0-429d-b36f-644069df51ed-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-szj4k\" (UID: \"6df88db0-10f0-429d-b36f-644069df51ed\") " pod="openshift-multus/multus-additional-cni-plugins-szj4k"
Sep 18 19:38:05.082226 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.080235    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b852eeb-95f7-40df-adbf-341896362824-etc-kubernetes\") pod \"multus-vxbw8\" (UID: \"2b852eeb-95f7-40df-adbf-341896362824\") " pod="openshift-multus/multus-vxbw8"
Sep 18 19:38:05.082226 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.080269    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2fd6f85b-2fd5-456a-919c-ca59d7e5203f-node-log\") pod \"ovnkube-node-xcbfz\" (UID: \"2fd6f85b-2fd5-456a-919c-ca59d7e5203f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz"
Sep 18 19:38:05.082226 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.080278    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f7e2771-a1bc-4ac9-9b67-1c7c9473d895-tmp-dir\") pod \"node-resolver-dwwf5\" (UID: \"3f7e2771-a1bc-4ac9-9b67-1c7c9473d895\") " pod="openshift-dns/node-resolver-dwwf5"
Sep 18 19:38:05.082226 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.080292    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/db9632d7-9a01-4dc2-a1ac-e1e7505e7b62-etc-modprobe-d\") pod \"tuned-96v5g\" (UID: \"db9632d7-9a01-4dc2-a1ac-e1e7505e7b62\") " pod="openshift-cluster-node-tuning-operator/tuned-96v5g"
Sep 18 19:38:05.082226 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.079814    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2b852eeb-95f7-40df-adbf-341896362824-multus-daemon-config\") pod \"multus-vxbw8\" (UID: \"2b852eeb-95f7-40df-adbf-341896362824\") " pod="openshift-multus/multus-vxbw8"
Sep 18 19:38:05.082226 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.080316    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/db9632d7-9a01-4dc2-a1ac-e1e7505e7b62-etc-sysconfig\") pod \"tuned-96v5g\" (UID: \"db9632d7-9a01-4dc2-a1ac-e1e7505e7b62\") " pod="openshift-cluster-node-tuning-operator/tuned-96v5g"
Sep 18 19:38:05.082226 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.080356    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/db9632d7-9a01-4dc2-a1ac-e1e7505e7b62-etc-tuned\") pod \"tuned-96v5g\" (UID: \"db9632d7-9a01-4dc2-a1ac-e1e7505e7b62\") " pod="openshift-cluster-node-tuning-operator/tuned-96v5g"
Sep 18 19:38:05.082226 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.080370    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/db9632d7-9a01-4dc2-a1ac-e1e7505e7b62-etc-sysconfig\") pod \"tuned-96v5g\" (UID: \"db9632d7-9a01-4dc2-a1ac-e1e7505e7b62\") " pod="openshift-cluster-node-tuning-operator/tuned-96v5g"
Sep 18 19:38:05.082748 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.080382    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2939ca32-81a7-4e9d-bdc6-ffca73820918-kubelet-config\") pod \"global-pull-secret-syncer-6vdpn\" (UID: \"2939ca32-81a7-4e9d-bdc6-ffca73820918\") " pod="kube-system/global-pull-secret-syncer-6vdpn"
Sep 18 19:38:05.082748 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.080394    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2fd6f85b-2fd5-456a-919c-ca59d7e5203f-node-log\") pod \"ovnkube-node-xcbfz\" (UID: \"2fd6f85b-2fd5-456a-919c-ca59d7e5203f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz"
Sep 18 19:38:05.082748 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.080459    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b852eeb-95f7-40df-adbf-341896362824-etc-kubernetes\") pod \"multus-vxbw8\" (UID: \"2b852eeb-95f7-40df-adbf-341896362824\") " pod="openshift-multus/multus-vxbw8"
Sep 18 19:38:05.082748 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.080466    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/db9632d7-9a01-4dc2-a1ac-e1e7505e7b62-etc-modprobe-d\") pod \"tuned-96v5g\" (UID: \"db9632d7-9a01-4dc2-a1ac-e1e7505e7b62\") " pod="openshift-cluster-node-tuning-operator/tuned-96v5g"
Sep 18 19:38:05.082748 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.080517    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2939ca32-81a7-4e9d-bdc6-ffca73820918-kubelet-config\") pod \"global-pull-secret-syncer-6vdpn\" (UID: \"2939ca32-81a7-4e9d-bdc6-ffca73820918\") " pod="kube-system/global-pull-secret-syncer-6vdpn"
Sep 18 19:38:05.082748 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.080595    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db9632d7-9a01-4dc2-a1ac-e1e7505e7b62-tmp\") pod \"tuned-96v5g\" (UID: \"db9632d7-9a01-4dc2-a1ac-e1e7505e7b62\") " pod="openshift-cluster-node-tuning-operator/tuned-96v5g"
Sep 18 19:38:05.082748 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.080608    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2fd6f85b-2fd5-456a-919c-ca59d7e5203f-env-overrides\") pod \"ovnkube-node-xcbfz\" (UID: \"2fd6f85b-2fd5-456a-919c-ca59d7e5203f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz"
Sep 18 19:38:05.082748 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.080594    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/71862b1f-4124-47a1-8942-60acbbb5bdfb-serviceca\") pod \"node-ca-8hdsx\" (UID: \"71862b1f-4124-47a1-8942-60acbbb5bdfb\") " pod="openshift-image-registry/node-ca-8hdsx"
Sep 18 19:38:05.082748 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.081077    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6df88db0-10f0-429d-b36f-644069df51ed-cni-binary-copy\") pod \"multus-additional-cni-plugins-szj4k\" (UID: \"6df88db0-10f0-429d-b36f-644069df51ed\") " pod="openshift-multus/multus-additional-cni-plugins-szj4k"
Sep 18 19:38:05.082748 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.081375    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2fd6f85b-2fd5-456a-919c-ca59d7e5203f-ovn-node-metrics-cert\") pod \"ovnkube-node-xcbfz\" (UID: \"2fd6f85b-2fd5-456a-919c-ca59d7e5203f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz"
Sep 18 19:38:05.082748 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.082217    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/db9632d7-9a01-4dc2-a1ac-e1e7505e7b62-etc-tuned\") pod \"tuned-96v5g\" (UID: \"db9632d7-9a01-4dc2-a1ac-e1e7505e7b62\") " pod="openshift-cluster-node-tuning-operator/tuned-96v5g"
Sep 18 19:38:05.083490 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.083473    2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:38:05.084839 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:05.084825    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:38:05.084880 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:05.084842    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:38:05.084880 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:05.084852    2564 projected.go:196] Error preparing data for projected volume kube-api-access-zjgc5 for pod openshift-network-diagnostics/network-check-target-857bd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:05.084937 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:05.084908    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b42bdad-4122-4a0a-978c-d2d5a7bdbd55-kube-api-access-zjgc5 podName:1b42bdad-4122-4a0a-978c-d2d5a7bdbd55 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:05.584895791 +0000 UTC m=+2.095251860 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-zjgc5" (UniqueName: "kubernetes.io/projected/1b42bdad-4122-4a0a-978c-d2d5a7bdbd55-kube-api-access-zjgc5") pod "network-check-target-857bd" (UID: "1b42bdad-4122-4a0a-978c-d2d5a7bdbd55") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:05.086994 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.086925    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5m8v\" (UniqueName: \"kubernetes.io/projected/6df88db0-10f0-429d-b36f-644069df51ed-kube-api-access-p5m8v\") pod \"multus-additional-cni-plugins-szj4k\" (UID: \"6df88db0-10f0-429d-b36f-644069df51ed\") " pod="openshift-multus/multus-additional-cni-plugins-szj4k"
Sep 18 19:38:05.087296 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.087272    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v67x6\" (UniqueName: \"kubernetes.io/projected/2fd6f85b-2fd5-456a-919c-ca59d7e5203f-kube-api-access-v67x6\") pod \"ovnkube-node-xcbfz\" (UID: \"2fd6f85b-2fd5-456a-919c-ca59d7e5203f\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz"
Sep 18 19:38:05.088120 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.088100    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7wk9\" (UniqueName: \"kubernetes.io/projected/71862b1f-4124-47a1-8942-60acbbb5bdfb-kube-api-access-d7wk9\") pod \"node-ca-8hdsx\" (UID: \"71862b1f-4124-47a1-8942-60acbbb5bdfb\") " pod="openshift-image-registry/node-ca-8hdsx"
Sep 18 19:38:05.088393 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.088375    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkpvv\" (UniqueName: \"kubernetes.io/projected/3f7e2771-a1bc-4ac9-9b67-1c7c9473d895-kube-api-access-pkpvv\") pod \"node-resolver-dwwf5\" (UID: \"3f7e2771-a1bc-4ac9-9b67-1c7c9473d895\") " pod="openshift-dns/node-resolver-dwwf5"
Sep 18 19:38:05.088544 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.088518    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8bwm\" (UniqueName: \"kubernetes.io/projected/937fca6a-d8cf-48b9-a0e5-39572043858e-kube-api-access-g8bwm\") pod \"iptables-alerter-tm2rm\" (UID: \"937fca6a-d8cf-48b9-a0e5-39572043858e\") " pod="openshift-network-operator/iptables-alerter-tm2rm"
Sep 18 19:38:05.089213 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.089187    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q2lc\" (UniqueName: \"kubernetes.io/projected/db9632d7-9a01-4dc2-a1ac-e1e7505e7b62-kube-api-access-2q2lc\") pod \"tuned-96v5g\" (UID: \"db9632d7-9a01-4dc2-a1ac-e1e7505e7b62\") " pod="openshift-cluster-node-tuning-operator/tuned-96v5g"
Sep 18 19:38:05.089298 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.089191    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5jfh\" (UniqueName: \"kubernetes.io/projected/2b852eeb-95f7-40df-adbf-341896362824-kube-api-access-f5jfh\") pod \"multus-vxbw8\" (UID: \"2b852eeb-95f7-40df-adbf-341896362824\") " pod="openshift-multus/multus-vxbw8"
Sep 18 19:38:05.182371 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.182296    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-667km"
Sep 18 19:38:05.193101 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:05.193073    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecb576b0_eafa_4997_9690_36f2ac243783.slice/crio-2811b0a0bc0d9b050312f47a1d16005c9fb37367c6b03548315ac82b4a18ca3a WatchSource:0}: Error finding container 2811b0a0bc0d9b050312f47a1d16005c9fb37367c6b03548315ac82b4a18ca3a: Status 404 returned error can't find the container with id 2811b0a0bc0d9b050312f47a1d16005c9fb37367c6b03548315ac82b4a18ca3a
Sep 18 19:38:05.196332 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.196309    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz"
Sep 18 19:38:05.204426 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:05.204400    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fd6f85b_2fd5_456a_919c_ca59d7e5203f.slice/crio-75a22905d60a2c87efd9c09c5a9ce8d5a05488e6ed552e480dc54c637951a1c3 WatchSource:0}: Error finding container 75a22905d60a2c87efd9c09c5a9ce8d5a05488e6ed552e480dc54c637951a1c3: Status 404 returned error can't find the container with id 75a22905d60a2c87efd9c09c5a9ce8d5a05488e6ed552e480dc54c637951a1c3
Sep 18 19:38:05.212352 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.212329    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47hjk"
Sep 18 19:38:05.216287 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.216266    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-szj4k"
Sep 18 19:38:05.221129 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.221108    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dwwf5"
Sep 18 19:38:05.222581 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:05.222546    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6d3d568_a824_4226_9944_66adc0973032.slice/crio-629d50e36bacf823b8f9adf5946917d2183dc868233a4131057eb0f35333901b WatchSource:0}: Error finding container 629d50e36bacf823b8f9adf5946917d2183dc868233a4131057eb0f35333901b: Status 404 returned error can't find the container with id 629d50e36bacf823b8f9adf5946917d2183dc868233a4131057eb0f35333901b
Sep 18 19:38:05.225902 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:05.225871    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6df88db0_10f0_429d_b36f_644069df51ed.slice/crio-5cbe0c20e1c8e5469a7c0a9eaf74ef6365486deb2807b57a25f0ce5021c16503 WatchSource:0}: Error finding container 5cbe0c20e1c8e5469a7c0a9eaf74ef6365486deb2807b57a25f0ce5021c16503: Status 404 returned error can't find the container with id 5cbe0c20e1c8e5469a7c0a9eaf74ef6365486deb2807b57a25f0ce5021c16503
Sep 18 19:38:05.231196 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:05.231172    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f7e2771_a1bc_4ac9_9b67_1c7c9473d895.slice/crio-ddc8eb66bc3e2fd4b58ff1e35848fcbf98159d5f4b7ddde1a281e71ad97cb289 WatchSource:0}: Error finding container ddc8eb66bc3e2fd4b58ff1e35848fcbf98159d5f4b7ddde1a281e71ad97cb289: Status 404 returned error can't find the container with id ddc8eb66bc3e2fd4b58ff1e35848fcbf98159d5f4b7ddde1a281e71ad97cb289
Sep 18 19:38:05.250604 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.250565    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8hdsx"
Sep 18 19:38:05.261614 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:05.261586    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71862b1f_4124_47a1_8942_60acbbb5bdfb.slice/crio-8a709dd5a5f04ac01384484cb685a39537ace3b94aad693fc9e66d01f812e64b WatchSource:0}: Error finding container 8a709dd5a5f04ac01384484cb685a39537ace3b94aad693fc9e66d01f812e64b: Status 404 returned error can't find the container with id 8a709dd5a5f04ac01384484cb685a39537ace3b94aad693fc9e66d01f812e64b
Sep 18 19:38:05.262290 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.262272    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vxbw8"
Sep 18 19:38:05.269090 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.269067    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tm2rm"
Sep 18 19:38:05.270451 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:05.270428    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b852eeb_95f7_40df_adbf_341896362824.slice/crio-411a4d8f26255c85131b393542136ed47bfbc537538d5a8cedec095f16a9983f WatchSource:0}: Error finding container 411a4d8f26255c85131b393542136ed47bfbc537538d5a8cedec095f16a9983f: Status 404 returned error can't find the container with id 411a4d8f26255c85131b393542136ed47bfbc537538d5a8cedec095f16a9983f
Sep 18 19:38:05.273645 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.273629    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-96v5g"
Sep 18 19:38:05.278238 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:05.278197    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod937fca6a_d8cf_48b9_a0e5_39572043858e.slice/crio-6392a39a16d2b57e2f3d8b6a1c5459d46a12f85f0bfb2d6befb31a2974d6750e WatchSource:0}: Error finding container 6392a39a16d2b57e2f3d8b6a1c5459d46a12f85f0bfb2d6befb31a2974d6750e: Status 404 returned error can't find the container with id 6392a39a16d2b57e2f3d8b6a1c5459d46a12f85f0bfb2d6befb31a2974d6750e
Sep 18 19:38:05.403627 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.403358    2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:38:05.484119 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.484032    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/256a72d8-e079-42fb-9426-c53adc283d1f-metrics-certs\") pod \"network-metrics-daemon-xgxbn\" (UID: \"256a72d8-e079-42fb-9426-c53adc283d1f\") " pod="openshift-multus/network-metrics-daemon-xgxbn"
Sep 18 19:38:05.484277 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:05.484172    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:05.484277 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:05.484240    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/256a72d8-e079-42fb-9426-c53adc283d1f-metrics-certs podName:256a72d8-e079-42fb-9426-c53adc283d1f nodeName:}" failed. No retries permitted until 2026-09-18 19:38:06.484221831 +0000 UTC m=+2.994577919 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/256a72d8-e079-42fb-9426-c53adc283d1f-metrics-certs") pod "network-metrics-daemon-xgxbn" (UID: "256a72d8-e079-42fb-9426-c53adc283d1f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:05.584926 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.584887    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2939ca32-81a7-4e9d-bdc6-ffca73820918-original-pull-secret\") pod \"global-pull-secret-syncer-6vdpn\" (UID: \"2939ca32-81a7-4e9d-bdc6-ffca73820918\") " pod="kube-system/global-pull-secret-syncer-6vdpn"
Sep 18 19:38:05.585130 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:05.585058    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:05.585195 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:05.585146    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2939ca32-81a7-4e9d-bdc6-ffca73820918-original-pull-secret podName:2939ca32-81a7-4e9d-bdc6-ffca73820918 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:06.585122475 +0000 UTC m=+3.095478567 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2939ca32-81a7-4e9d-bdc6-ffca73820918-original-pull-secret") pod "global-pull-secret-syncer-6vdpn" (UID: "2939ca32-81a7-4e9d-bdc6-ffca73820918") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:05.628473 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.628249    2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:38:05.686139 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.685795    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjgc5\" (UniqueName: \"kubernetes.io/projected/1b42bdad-4122-4a0a-978c-d2d5a7bdbd55-kube-api-access-zjgc5\") pod \"network-check-target-857bd\" (UID: \"1b42bdad-4122-4a0a-978c-d2d5a7bdbd55\") " pod="openshift-network-diagnostics/network-check-target-857bd"
Sep 18 19:38:05.686139 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:05.686017    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:38:05.686139 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:05.686042    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:38:05.686139 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:05.686055    2564 projected.go:196] Error preparing data for projected volume kube-api-access-zjgc5 for pod openshift-network-diagnostics/network-check-target-857bd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:05.686139 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:05.686115    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b42bdad-4122-4a0a-978c-d2d5a7bdbd55-kube-api-access-zjgc5 podName:1b42bdad-4122-4a0a-978c-d2d5a7bdbd55 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:06.686096169 +0000 UTC m=+3.196452243 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-zjgc5" (UniqueName: "kubernetes.io/projected/1b42bdad-4122-4a0a-978c-d2d5a7bdbd55-kube-api-access-zjgc5") pod "network-check-target-857bd" (UID: "1b42bdad-4122-4a0a-978c-d2d5a7bdbd55") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:05.912927 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.912812    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 19:33:05 +0000 UTC" deadline="2028-05-28 05:52:12.036253336 +0000 UTC"
Sep 18 19:38:05.912927 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:05.912875    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14818h14m6.123382115s"
Sep 18 19:38:06.033414 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:06.033367    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-96v5g" event={"ID":"db9632d7-9a01-4dc2-a1ac-e1e7505e7b62","Type":"ContainerStarted","Data":"316ab3a5f57a708a9b9b5341ed15fdd80d7c3a9370ba43f8e9d45161e30b544e"}
Sep 18 19:38:06.063318 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:06.063280    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vxbw8" event={"ID":"2b852eeb-95f7-40df-adbf-341896362824","Type":"ContainerStarted","Data":"411a4d8f26255c85131b393542136ed47bfbc537538d5a8cedec095f16a9983f"}
Sep 18 19:38:06.090950 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:06.090661    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-szj4k" event={"ID":"6df88db0-10f0-429d-b36f-644069df51ed","Type":"ContainerStarted","Data":"5cbe0c20e1c8e5469a7c0a9eaf74ef6365486deb2807b57a25f0ce5021c16503"}
Sep 18 19:38:06.113273 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:06.113231    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47hjk" event={"ID":"f6d3d568-a824-4226-9944-66adc0973032","Type":"ContainerStarted","Data":"629d50e36bacf823b8f9adf5946917d2183dc868233a4131057eb0f35333901b"}
Sep 18 19:38:06.132213 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:06.132121    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz" event={"ID":"2fd6f85b-2fd5-456a-919c-ca59d7e5203f","Type":"ContainerStarted","Data":"75a22905d60a2c87efd9c09c5a9ce8d5a05488e6ed552e480dc54c637951a1c3"}
Sep 18 19:38:06.142976 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:06.142930    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tm2rm" event={"ID":"937fca6a-d8cf-48b9-a0e5-39572043858e","Type":"ContainerStarted","Data":"6392a39a16d2b57e2f3d8b6a1c5459d46a12f85f0bfb2d6befb31a2974d6750e"}
Sep 18 19:38:06.161180 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:06.161139    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8hdsx" event={"ID":"71862b1f-4124-47a1-8942-60acbbb5bdfb","Type":"ContainerStarted","Data":"8a709dd5a5f04ac01384484cb685a39537ace3b94aad693fc9e66d01f812e64b"}
Sep 18 19:38:06.173790 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:06.173693    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dwwf5" event={"ID":"3f7e2771-a1bc-4ac9-9b67-1c7c9473d895","Type":"ContainerStarted","Data":"ddc8eb66bc3e2fd4b58ff1e35848fcbf98159d5f4b7ddde1a281e71ad97cb289"}
Sep 18 19:38:06.187341 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:06.187302    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-667km" event={"ID":"ecb576b0-eafa-4997-9690-36f2ac243783","Type":"ContainerStarted","Data":"2811b0a0bc0d9b050312f47a1d16005c9fb37367c6b03548315ac82b4a18ca3a"}
Sep 18 19:38:06.492303 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:06.492206    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/256a72d8-e079-42fb-9426-c53adc283d1f-metrics-certs\") pod \"network-metrics-daemon-xgxbn\" (UID: \"256a72d8-e079-42fb-9426-c53adc283d1f\") " pod="openshift-multus/network-metrics-daemon-xgxbn"
Sep 18 19:38:06.492490 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:06.492389    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:06.492490 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:06.492475    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/256a72d8-e079-42fb-9426-c53adc283d1f-metrics-certs podName:256a72d8-e079-42fb-9426-c53adc283d1f nodeName:}" failed. No retries permitted until 2026-09-18 19:38:08.49244996 +0000 UTC m=+5.002806053 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/256a72d8-e079-42fb-9426-c53adc283d1f-metrics-certs") pod "network-metrics-daemon-xgxbn" (UID: "256a72d8-e079-42fb-9426-c53adc283d1f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:06.592920 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:06.592878    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2939ca32-81a7-4e9d-bdc6-ffca73820918-original-pull-secret\") pod \"global-pull-secret-syncer-6vdpn\" (UID: \"2939ca32-81a7-4e9d-bdc6-ffca73820918\") " pod="kube-system/global-pull-secret-syncer-6vdpn"
Sep 18 19:38:06.593146 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:06.593102    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:06.593212 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:06.593175    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2939ca32-81a7-4e9d-bdc6-ffca73820918-original-pull-secret podName:2939ca32-81a7-4e9d-bdc6-ffca73820918 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:08.593155518 +0000 UTC m=+5.103511607 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2939ca32-81a7-4e9d-bdc6-ffca73820918-original-pull-secret") pod "global-pull-secret-syncer-6vdpn" (UID: "2939ca32-81a7-4e9d-bdc6-ffca73820918") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:06.694013 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:06.693965    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjgc5\" (UniqueName: \"kubernetes.io/projected/1b42bdad-4122-4a0a-978c-d2d5a7bdbd55-kube-api-access-zjgc5\") pod \"network-check-target-857bd\" (UID: \"1b42bdad-4122-4a0a-978c-d2d5a7bdbd55\") " pod="openshift-network-diagnostics/network-check-target-857bd"
Sep 18 19:38:06.694288 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:06.694260    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:38:06.694288 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:06.694286    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:38:06.694417 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:06.694301    2564 projected.go:196] Error preparing data for projected volume kube-api-access-zjgc5 for pod openshift-network-diagnostics/network-check-target-857bd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:06.694417 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:06.694368    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b42bdad-4122-4a0a-978c-d2d5a7bdbd55-kube-api-access-zjgc5 podName:1b42bdad-4122-4a0a-978c-d2d5a7bdbd55 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:08.694347896 +0000 UTC m=+5.204703970 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-zjgc5" (UniqueName: "kubernetes.io/projected/1b42bdad-4122-4a0a-978c-d2d5a7bdbd55-kube-api-access-zjgc5") pod "network-check-target-857bd" (UID: "1b42bdad-4122-4a0a-978c-d2d5a7bdbd55") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:06.913841 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:06.913726    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 19:33:05 +0000 UTC" deadline="2028-04-24 05:51:39.154859895 +0000 UTC"
Sep 18 19:38:06.913841 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:06.913770    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14002h13m32.241095138s"
Sep 18 19:38:06.993814 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:06.993785    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vdpn"
Sep 18 19:38:06.994019 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:06.993922    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vdpn" podUID="2939ca32-81a7-4e9d-bdc6-ffca73820918"
Sep 18 19:38:06.994435 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:06.994412    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xgxbn"
Sep 18 19:38:06.994559 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:06.994533    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xgxbn" podUID="256a72d8-e079-42fb-9426-c53adc283d1f"
Sep 18 19:38:06.994626 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:06.994608    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-857bd"
Sep 18 19:38:06.994731 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:06.994688    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-857bd" podUID="1b42bdad-4122-4a0a-978c-d2d5a7bdbd55"
Sep 18 19:38:07.345183 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:07.345085    2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:38:08.512859 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:08.512788    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/256a72d8-e079-42fb-9426-c53adc283d1f-metrics-certs\") pod \"network-metrics-daemon-xgxbn\" (UID: \"256a72d8-e079-42fb-9426-c53adc283d1f\") " pod="openshift-multus/network-metrics-daemon-xgxbn"
Sep 18 19:38:08.513458 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:08.513057    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:08.513458 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:08.513137    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/256a72d8-e079-42fb-9426-c53adc283d1f-metrics-certs podName:256a72d8-e079-42fb-9426-c53adc283d1f nodeName:}" failed. No retries permitted until 2026-09-18 19:38:12.513115623 +0000 UTC m=+9.023471712 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/256a72d8-e079-42fb-9426-c53adc283d1f-metrics-certs") pod "network-metrics-daemon-xgxbn" (UID: "256a72d8-e079-42fb-9426-c53adc283d1f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:08.614078 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:08.613433    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2939ca32-81a7-4e9d-bdc6-ffca73820918-original-pull-secret\") pod \"global-pull-secret-syncer-6vdpn\" (UID: \"2939ca32-81a7-4e9d-bdc6-ffca73820918\") " pod="kube-system/global-pull-secret-syncer-6vdpn"
Sep 18 19:38:08.614078 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:08.613623    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:08.614078 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:08.613694    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2939ca32-81a7-4e9d-bdc6-ffca73820918-original-pull-secret podName:2939ca32-81a7-4e9d-bdc6-ffca73820918 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:12.613673544 +0000 UTC m=+9.124029636 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2939ca32-81a7-4e9d-bdc6-ffca73820918-original-pull-secret") pod "global-pull-secret-syncer-6vdpn" (UID: "2939ca32-81a7-4e9d-bdc6-ffca73820918") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:08.714785 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:08.714012    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjgc5\" (UniqueName: \"kubernetes.io/projected/1b42bdad-4122-4a0a-978c-d2d5a7bdbd55-kube-api-access-zjgc5\") pod \"network-check-target-857bd\" (UID: \"1b42bdad-4122-4a0a-978c-d2d5a7bdbd55\") " pod="openshift-network-diagnostics/network-check-target-857bd"
Sep 18 19:38:08.714785 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:08.714307    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:38:08.714785 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:08.714334    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:38:08.714785 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:08.714350    2564 projected.go:196] Error preparing data for projected volume kube-api-access-zjgc5 for pod openshift-network-diagnostics/network-check-target-857bd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:08.714785 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:08.714415    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b42bdad-4122-4a0a-978c-d2d5a7bdbd55-kube-api-access-zjgc5 podName:1b42bdad-4122-4a0a-978c-d2d5a7bdbd55 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:12.714393977 +0000 UTC m=+9.224750073 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-zjgc5" (UniqueName: "kubernetes.io/projected/1b42bdad-4122-4a0a-978c-d2d5a7bdbd55-kube-api-access-zjgc5") pod "network-check-target-857bd" (UID: "1b42bdad-4122-4a0a-978c-d2d5a7bdbd55") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:08.994001 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:08.993888    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vdpn"
Sep 18 19:38:08.994170 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:08.994050    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vdpn" podUID="2939ca32-81a7-4e9d-bdc6-ffca73820918"
Sep 18 19:38:08.994395 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:08.994365    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-857bd"
Sep 18 19:38:08.994509 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:08.994485    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-857bd" podUID="1b42bdad-4122-4a0a-978c-d2d5a7bdbd55"
Sep 18 19:38:08.994572 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:08.994510    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xgxbn"
Sep 18 19:38:08.994687 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:08.994648    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xgxbn" podUID="256a72d8-e079-42fb-9426-c53adc283d1f"
Sep 18 19:38:10.994332 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:10.993757    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vdpn"
Sep 18 19:38:10.994332 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:10.993900    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xgxbn"
Sep 18 19:38:10.994332 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:10.993916    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vdpn" podUID="2939ca32-81a7-4e9d-bdc6-ffca73820918"
Sep 18 19:38:10.994332 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:10.994082    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xgxbn" podUID="256a72d8-e079-42fb-9426-c53adc283d1f"
Sep 18 19:38:10.994332 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:10.994156    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-857bd"
Sep 18 19:38:10.994332 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:10.994236    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-857bd" podUID="1b42bdad-4122-4a0a-978c-d2d5a7bdbd55"
Sep 18 19:38:12.553056 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:12.552982    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/256a72d8-e079-42fb-9426-c53adc283d1f-metrics-certs\") pod \"network-metrics-daemon-xgxbn\" (UID: \"256a72d8-e079-42fb-9426-c53adc283d1f\") " pod="openshift-multus/network-metrics-daemon-xgxbn"
Sep 18 19:38:12.553710 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:12.553199    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:12.553710 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:12.553285    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/256a72d8-e079-42fb-9426-c53adc283d1f-metrics-certs podName:256a72d8-e079-42fb-9426-c53adc283d1f nodeName:}" failed. No retries permitted until 2026-09-18 19:38:20.553262299 +0000 UTC m=+17.063618386 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/256a72d8-e079-42fb-9426-c53adc283d1f-metrics-certs") pod "network-metrics-daemon-xgxbn" (UID: "256a72d8-e079-42fb-9426-c53adc283d1f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:12.654203 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:12.654150    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2939ca32-81a7-4e9d-bdc6-ffca73820918-original-pull-secret\") pod \"global-pull-secret-syncer-6vdpn\" (UID: \"2939ca32-81a7-4e9d-bdc6-ffca73820918\") " pod="kube-system/global-pull-secret-syncer-6vdpn"
Sep 18 19:38:12.654410 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:12.654330    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:12.654410 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:12.654403    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2939ca32-81a7-4e9d-bdc6-ffca73820918-original-pull-secret podName:2939ca32-81a7-4e9d-bdc6-ffca73820918 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:20.654383143 +0000 UTC m=+17.164739229 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2939ca32-81a7-4e9d-bdc6-ffca73820918-original-pull-secret") pod "global-pull-secret-syncer-6vdpn" (UID: "2939ca32-81a7-4e9d-bdc6-ffca73820918") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:12.754926 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:12.754891    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjgc5\" (UniqueName: \"kubernetes.io/projected/1b42bdad-4122-4a0a-978c-d2d5a7bdbd55-kube-api-access-zjgc5\") pod \"network-check-target-857bd\" (UID: \"1b42bdad-4122-4a0a-978c-d2d5a7bdbd55\") " pod="openshift-network-diagnostics/network-check-target-857bd"
Sep 18 19:38:12.755188 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:12.755107    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:38:12.755188 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:12.755174    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:38:12.755312 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:12.755192    2564 projected.go:196] Error preparing data for projected volume kube-api-access-zjgc5 for pod openshift-network-diagnostics/network-check-target-857bd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:12.755312 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:12.755266    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b42bdad-4122-4a0a-978c-d2d5a7bdbd55-kube-api-access-zjgc5 podName:1b42bdad-4122-4a0a-978c-d2d5a7bdbd55 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:20.755244607 +0000 UTC m=+17.265600682 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-zjgc5" (UniqueName: "kubernetes.io/projected/1b42bdad-4122-4a0a-978c-d2d5a7bdbd55-kube-api-access-zjgc5") pod "network-check-target-857bd" (UID: "1b42bdad-4122-4a0a-978c-d2d5a7bdbd55") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:12.994034 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:12.993937    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vdpn"
Sep 18 19:38:12.994034 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:12.993991    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xgxbn"
Sep 18 19:38:12.994252 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:12.994105    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vdpn" podUID="2939ca32-81a7-4e9d-bdc6-ffca73820918"
Sep 18 19:38:12.994618 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:12.994584    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xgxbn" podUID="256a72d8-e079-42fb-9426-c53adc283d1f"
Sep 18 19:38:12.994714 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:12.994644    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-857bd"
Sep 18 19:38:12.994771 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:12.994714    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-857bd" podUID="1b42bdad-4122-4a0a-978c-d2d5a7bdbd55"
Sep 18 19:38:14.994040 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:14.993932    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vdpn"
Sep 18 19:38:14.994040 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:14.993932    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-857bd"
Sep 18 19:38:14.994497 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:14.994093    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xgxbn"
Sep 18 19:38:14.994497 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:14.994083    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vdpn" podUID="2939ca32-81a7-4e9d-bdc6-ffca73820918"
Sep 18 19:38:14.994497 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:14.994190    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-857bd" podUID="1b42bdad-4122-4a0a-978c-d2d5a7bdbd55"
Sep 18 19:38:14.994497 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:14.994309    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xgxbn" podUID="256a72d8-e079-42fb-9426-c53adc283d1f"
Sep 18 19:38:16.994237 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:16.994201    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xgxbn"
Sep 18 19:38:16.994622 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:16.994214    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-857bd"
Sep 18 19:38:16.994622 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:16.994314    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xgxbn" podUID="256a72d8-e079-42fb-9426-c53adc283d1f"
Sep 18 19:38:16.994622 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:16.994322    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vdpn"
Sep 18 19:38:16.994622 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:16.994418    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-857bd" podUID="1b42bdad-4122-4a0a-978c-d2d5a7bdbd55"
Sep 18 19:38:16.994622 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:16.994483    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vdpn" podUID="2939ca32-81a7-4e9d-bdc6-ffca73820918"
Sep 18 19:38:18.994158 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:18.994041    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vdpn"
Sep 18 19:38:18.994674 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:18.994179    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vdpn" podUID="2939ca32-81a7-4e9d-bdc6-ffca73820918"
Sep 18 19:38:18.994674 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:18.994050    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-857bd"
Sep 18 19:38:18.994674 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:18.994050    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xgxbn"
Sep 18 19:38:18.994674 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:18.994281    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-857bd" podUID="1b42bdad-4122-4a0a-978c-d2d5a7bdbd55"
Sep 18 19:38:18.994674 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:18.994365    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xgxbn" podUID="256a72d8-e079-42fb-9426-c53adc283d1f"
Sep 18 19:38:20.615819 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:20.615780    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/256a72d8-e079-42fb-9426-c53adc283d1f-metrics-certs\") pod \"network-metrics-daemon-xgxbn\" (UID: \"256a72d8-e079-42fb-9426-c53adc283d1f\") " pod="openshift-multus/network-metrics-daemon-xgxbn"
Sep 18 19:38:20.616420 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:20.615998    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:20.616420 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:20.616082    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/256a72d8-e079-42fb-9426-c53adc283d1f-metrics-certs podName:256a72d8-e079-42fb-9426-c53adc283d1f nodeName:}" failed. No retries permitted until 2026-09-18 19:38:36.616059185 +0000 UTC m=+33.126415270 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/256a72d8-e079-42fb-9426-c53adc283d1f-metrics-certs") pod "network-metrics-daemon-xgxbn" (UID: "256a72d8-e079-42fb-9426-c53adc283d1f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:20.716502 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:20.716457    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2939ca32-81a7-4e9d-bdc6-ffca73820918-original-pull-secret\") pod \"global-pull-secret-syncer-6vdpn\" (UID: \"2939ca32-81a7-4e9d-bdc6-ffca73820918\") " pod="kube-system/global-pull-secret-syncer-6vdpn"
Sep 18 19:38:20.716681 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:20.716649    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:20.716751 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:20.716738    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2939ca32-81a7-4e9d-bdc6-ffca73820918-original-pull-secret podName:2939ca32-81a7-4e9d-bdc6-ffca73820918 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:36.716716293 +0000 UTC m=+33.227072376 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2939ca32-81a7-4e9d-bdc6-ffca73820918-original-pull-secret") pod "global-pull-secret-syncer-6vdpn" (UID: "2939ca32-81a7-4e9d-bdc6-ffca73820918") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:20.817075 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:20.817031    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjgc5\" (UniqueName: \"kubernetes.io/projected/1b42bdad-4122-4a0a-978c-d2d5a7bdbd55-kube-api-access-zjgc5\") pod \"network-check-target-857bd\" (UID: \"1b42bdad-4122-4a0a-978c-d2d5a7bdbd55\") " pod="openshift-network-diagnostics/network-check-target-857bd"
Sep 18 19:38:20.817273 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:20.817248    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:38:20.817333 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:20.817287    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:38:20.817333 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:20.817302    2564 projected.go:196] Error preparing data for projected volume kube-api-access-zjgc5 for pod openshift-network-diagnostics/network-check-target-857bd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:20.817417 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:20.817373    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b42bdad-4122-4a0a-978c-d2d5a7bdbd55-kube-api-access-zjgc5 podName:1b42bdad-4122-4a0a-978c-d2d5a7bdbd55 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:36.817354654 +0000 UTC m=+33.327710746 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-zjgc5" (UniqueName: "kubernetes.io/projected/1b42bdad-4122-4a0a-978c-d2d5a7bdbd55-kube-api-access-zjgc5") pod "network-check-target-857bd" (UID: "1b42bdad-4122-4a0a-978c-d2d5a7bdbd55") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:20.993891 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:20.993810    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-857bd"
Sep 18 19:38:20.994092 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:20.993814    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vdpn"
Sep 18 19:38:20.994092 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:20.993924    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-857bd" podUID="1b42bdad-4122-4a0a-978c-d2d5a7bdbd55"
Sep 18 19:38:20.994092 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:20.994041    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vdpn" podUID="2939ca32-81a7-4e9d-bdc6-ffca73820918"
Sep 18 19:38:20.994092 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:20.993829    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xgxbn"
Sep 18 19:38:20.994258 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:20.994141    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xgxbn" podUID="256a72d8-e079-42fb-9426-c53adc283d1f"
Sep 18 19:38:22.993409 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:22.993369    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vdpn"
Sep 18 19:38:22.993409 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:22.993394    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xgxbn"
Sep 18 19:38:22.993908 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:22.993530    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-857bd"
Sep 18 19:38:22.993908 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:22.993537    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vdpn" podUID="2939ca32-81a7-4e9d-bdc6-ffca73820918"
Sep 18 19:38:22.996423 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:22.993946    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xgxbn" podUID="256a72d8-e079-42fb-9426-c53adc283d1f"
Sep 18 19:38:22.996423 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:22.994928    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-857bd" podUID="1b42bdad-4122-4a0a-978c-d2d5a7bdbd55"
Sep 18 19:38:24.993506 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:24.993478    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vdpn"
Sep 18 19:38:24.994026 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:24.993493    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-857bd"
Sep 18 19:38:24.994026 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:24.993610    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vdpn" podUID="2939ca32-81a7-4e9d-bdc6-ffca73820918"
Sep 18 19:38:24.994026 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:24.993493    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xgxbn"
Sep 18 19:38:24.994026 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:24.993651    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-857bd" podUID="1b42bdad-4122-4a0a-978c-d2d5a7bdbd55"
Sep 18 19:38:24.994026 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:24.993745    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xgxbn" podUID="256a72d8-e079-42fb-9426-c53adc283d1f"
Sep 18 19:38:25.247006 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:25.245808    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-161.ec2.internal" event={"ID":"9eb8c7aafdc391f2f7ae0f04a98530ea","Type":"ContainerStarted","Data":"bd033b5e71d0da280fa28450383ed47b05efbe9f3d2bd8ad6916785a1e8dbeab"}
Sep 18 19:38:25.253759 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:25.253728    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-96v5g" event={"ID":"db9632d7-9a01-4dc2-a1ac-e1e7505e7b62","Type":"ContainerStarted","Data":"47ce4f4972d6187809ecc8903030a4d492c3790beecfeca1d6e6d60ebd6b25b7"}
Sep 18 19:38:25.255769 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:25.255748    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vxbw8" event={"ID":"2b852eeb-95f7-40df-adbf-341896362824","Type":"ContainerStarted","Data":"e9ec659f9ef4a0feb5dbbf24e46204298e2d8931d1d4bc1d9a51c719d971c944"}
Sep 18 19:38:25.259047 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:25.258933    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-161.ec2.internal" podStartSLOduration=20.258913542 podStartE2EDuration="20.258913542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:38:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:25.258825187 +0000 UTC m=+21.769181303" watchObservedRunningTime="2026-09-18 19:38:25.258913542 +0000 UTC m=+21.769269662"
Sep 18 19:38:25.278680 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:25.276672    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-vxbw8" podStartSLOduration=1.557044489 podStartE2EDuration="21.276651059s" totalImagesPullingTime="19.71960657s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:38:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:25.276100234 +0000 UTC m=+21.786456322" watchObservedRunningTime="2026-09-18 19:38:25.276651059 +0000 UTC m=+21.787007150"
Sep 18 19:38:26.260199 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:26.259853    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8hdsx" event={"ID":"71862b1f-4124-47a1-8942-60acbbb5bdfb","Type":"ContainerStarted","Data":"66e18a50d7b8275ae12d1fd5204c8b3a6d4e1839dbaca526b1fee669cac68002"}
Sep 18 19:38:26.261782 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:26.261721    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dwwf5" event={"ID":"3f7e2771-a1bc-4ac9-9b67-1c7c9473d895","Type":"ContainerStarted","Data":"632218b63e7796828cb56d2ded0d56be3eca5c29cc630ed8de41bb8a4bf564d1"}
Sep 18 19:38:26.263554 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:26.263524    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-667km" event={"ID":"ecb576b0-eafa-4997-9690-36f2ac243783","Type":"ContainerStarted","Data":"a43ea677f2eac6cb897a0e7254a48573908ffb578a605f3dcb607d7a8be7e605"}
Sep 18 19:38:26.265432 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:26.265402    2564 generic.go:356] "Generic (PLEG): container finished" podID="6df88db0-10f0-429d-b36f-644069df51ed" containerID="c8aa03f4b058699c513123a4c39b2cd7140e37ec004b4bb81f71590cfdf6c681" exitCode=0
Sep 18 19:38:26.265563 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:26.265487    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-szj4k" event={"ID":"6df88db0-10f0-429d-b36f-644069df51ed","Type":"ContainerDied","Data":"c8aa03f4b058699c513123a4c39b2cd7140e37ec004b4bb81f71590cfdf6c681"}
Sep 18 19:38:26.267303 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:26.267283    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47hjk" event={"ID":"f6d3d568-a824-4226-9944-66adc0973032","Type":"ContainerStarted","Data":"305ec26e9034149987e231edd4c86ade030a9ddaede66ffa08e0cfa106da3e93"}
Sep 18 19:38:26.270802 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:26.270779    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz" event={"ID":"2fd6f85b-2fd5-456a-919c-ca59d7e5203f","Type":"ContainerStarted","Data":"43f462e523e9b430d862b6ea61473356950a7a70f70108f5c744617f856f611b"}
Sep 18 19:38:26.270912 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:26.270843    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz" event={"ID":"2fd6f85b-2fd5-456a-919c-ca59d7e5203f","Type":"ContainerStarted","Data":"7a784738820f8ca7e74df95b59f5c5a045cdcb27c2dd22082d8d4e4a58730095"}
Sep 18 19:38:26.270912 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:26.270873    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz" event={"ID":"2fd6f85b-2fd5-456a-919c-ca59d7e5203f","Type":"ContainerStarted","Data":"3811fe118b981b6f6329feac6b0eeb42f9346e88d59984c0eb9760f032c28d45"}
Sep 18 19:38:26.270912 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:26.270902    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz" event={"ID":"2fd6f85b-2fd5-456a-919c-ca59d7e5203f","Type":"ContainerStarted","Data":"3f4134b5064f437e04bf9165a9b20dbc5814df5d147eef39d2b7937cbe2721ab"}
Sep 18 19:38:26.271090 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:26.270939    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz" event={"ID":"2fd6f85b-2fd5-456a-919c-ca59d7e5203f","Type":"ContainerStarted","Data":"3e2e0c321dbb004e3ef980e36817dcbc2add482189d6006ea985bc926e2f5111"}
Sep 18 19:38:26.271090 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:26.270974    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz" event={"ID":"2fd6f85b-2fd5-456a-919c-ca59d7e5203f","Type":"ContainerStarted","Data":"a8606c62a8becc7106bd13a7e8b683f27e80bdd5b95c581e49ca23489a4542ed"}
Sep 18 19:38:26.273016 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:26.272977    2564 generic.go:356] "Generic (PLEG): container finished" podID="2c1e9c7cacf21e14413fce157c96f3cd" containerID="600417a5e92ed85ceb5d935f1ee013bd55c6c084245021bc8fea1c5f7a3d1aab" exitCode=0
Sep 18 19:38:26.273132 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:26.273095    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.ec2.internal" event={"ID":"2c1e9c7cacf21e14413fce157c96f3cd","Type":"ContainerDied","Data":"600417a5e92ed85ceb5d935f1ee013bd55c6c084245021bc8fea1c5f7a3d1aab"}
Sep 18 19:38:26.278333 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:26.278280    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-96v5g" podStartSLOduration=2.609572751 podStartE2EDuration="22.278261344s" totalImagesPullingTime="19.668688593s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:38:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:25.293337015 +0000 UTC m=+21.803693105" watchObservedRunningTime="2026-09-18 19:38:26.278261344 +0000 UTC m=+22.788617436"
Sep 18 19:38:26.291649 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:26.291593    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-8hdsx" podStartSLOduration=2.6368313370000003 podStartE2EDuration="22.291578079s" totalImagesPullingTime="19.654746742s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:38:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:26.275790141 +0000 UTC m=+22.786146251" watchObservedRunningTime="2026-09-18 19:38:26.291578079 +0000 UTC m=+22.801934169"
Sep 18 19:38:26.316357 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:26.316301    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-dwwf5" podStartSLOduration=2.869678429 podStartE2EDuration="22.316279465s" totalImagesPullingTime="19.446601036s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:38:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:26.299284378 +0000 UTC m=+22.809640490" watchObservedRunningTime="2026-09-18 19:38:26.316279465 +0000 UTC m=+22.826635557"
Sep 18 19:38:26.316772 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:26.316737    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-667km" podStartSLOduration=6.509558056 podStartE2EDuration="22.31672735s" totalImagesPullingTime="15.807169294s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:38:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:26.315998606 +0000 UTC m=+22.826354723" watchObservedRunningTime="2026-09-18 19:38:26.31672735 +0000 UTC m=+22.827083440"
Sep 18 19:38:26.724833 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:26.724790    2564 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 18 19:38:26.942048 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:26.941660    2564 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-18T19:38:26.724811641Z","UUID":"36ba6bb0-0cdf-49b9-b223-ee3a3948374b","Handler":null,"Name":"","Endpoint":""}
Sep 18 19:38:26.943254 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:26.943230    2564 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 18 19:38:26.943388 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:26.943268    2564 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 18 19:38:26.994099 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:26.994064    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vdpn"
Sep 18 19:38:26.994099 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:26.994087    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-857bd"
Sep 18 19:38:26.994337 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:26.994087    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xgxbn"
Sep 18 19:38:26.994337 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:26.994196    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vdpn" podUID="2939ca32-81a7-4e9d-bdc6-ffca73820918"
Sep 18 19:38:26.994337 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:26.994274    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-857bd" podUID="1b42bdad-4122-4a0a-978c-d2d5a7bdbd55"
Sep 18 19:38:26.994483 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:26.994362    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xgxbn" podUID="256a72d8-e079-42fb-9426-c53adc283d1f"
Sep 18 19:38:27.210730 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:27.210649    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-667km"
Sep 18 19:38:27.211206 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:27.211182    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-667km"
Sep 18 19:38:27.277966 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:27.277930    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47hjk" event={"ID":"f6d3d568-a824-4226-9944-66adc0973032","Type":"ContainerStarted","Data":"2d92683fc62ecc143a4118c74278100d9ec48ef366cd9cccc37f55777fc0fc86"}
Sep 18 19:38:27.280729 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:27.280698    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.ec2.internal" event={"ID":"2c1e9c7cacf21e14413fce157c96f3cd","Type":"ContainerStarted","Data":"800d969869672ae323b9932132f3fec6abaf7c37d0bee422e02160e1f2ef274b"}
Sep 18 19:38:27.283332 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:27.283259    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tm2rm" event={"ID":"937fca6a-d8cf-48b9-a0e5-39572043858e","Type":"ContainerStarted","Data":"b1c98539f0cda0de2b4d8394c551430c4461f9733b375bfc3ff000ca903a9e70"}
Sep 18 19:38:27.296159 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:27.296099    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.ec2.internal" podStartSLOduration=22.296082785 podStartE2EDuration="22.296082785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:38:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:27.295486047 +0000 UTC m=+23.805842148" watchObservedRunningTime="2026-09-18 19:38:27.296082785 +0000 UTC m=+23.806438875"
Sep 18 19:38:27.308814 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:27.308754    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-tm2rm" podStartSLOduration=3.634036541 podStartE2EDuration="23.308738122s" totalImagesPullingTime="19.674701581s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:38:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:27.308118185 +0000 UTC m=+23.818474334" watchObservedRunningTime="2026-09-18 19:38:27.308738122 +0000 UTC m=+23.819094213"
Sep 18 19:38:28.286764 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:28.286679    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47hjk" event={"ID":"f6d3d568-a824-4226-9944-66adc0973032","Type":"ContainerStarted","Data":"34ba13b36b800ebb375e74126255edd2600d69e309a8cfb03fa6a7ac9087e678"}
Sep 18 19:38:28.290205 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:28.290182    2564 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 18 19:38:28.290205 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:28.290188    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz" event={"ID":"2fd6f85b-2fd5-456a-919c-ca59d7e5203f","Type":"ContainerStarted","Data":"2aaffaa4ec17beb8e790775b4bf79142db31f4cddcd82d15a0153d53479e08eb"}
Sep 18 19:38:28.993698 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:28.993657    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vdpn"
Sep 18 19:38:28.993698 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:28.993683    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xgxbn"
Sep 18 19:38:28.994006 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:28.993775    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-857bd"
Sep 18 19:38:28.994006 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:28.993788    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vdpn" podUID="2939ca32-81a7-4e9d-bdc6-ffca73820918"
Sep 18 19:38:28.994006 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:28.993894    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-857bd" podUID="1b42bdad-4122-4a0a-978c-d2d5a7bdbd55"
Sep 18 19:38:28.994160 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:28.994011    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xgxbn" podUID="256a72d8-e079-42fb-9426-c53adc283d1f"
Sep 18 19:38:30.994237 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:30.994012    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vdpn"
Sep 18 19:38:30.994650 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:30.994063    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-857bd"
Sep 18 19:38:30.994650 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:30.994259    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vdpn" podUID="2939ca32-81a7-4e9d-bdc6-ffca73820918"
Sep 18 19:38:30.994650 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:30.994357    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-857bd" podUID="1b42bdad-4122-4a0a-978c-d2d5a7bdbd55"
Sep 18 19:38:30.994650 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:30.994063    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xgxbn"
Sep 18 19:38:30.994650 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:30.994473    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xgxbn" podUID="256a72d8-e079-42fb-9426-c53adc283d1f"
Sep 18 19:38:31.298960 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:31.298866    2564 generic.go:356] "Generic (PLEG): container finished" podID="6df88db0-10f0-429d-b36f-644069df51ed" containerID="3a53d2bdd09be19202a59ecfe583e5f07f33a6067a2e33bb49bb84c63b5ded08" exitCode=0
Sep 18 19:38:31.298960 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:31.298919    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-szj4k" event={"ID":"6df88db0-10f0-429d-b36f-644069df51ed","Type":"ContainerDied","Data":"3a53d2bdd09be19202a59ecfe583e5f07f33a6067a2e33bb49bb84c63b5ded08"}
Sep 18 19:38:31.303226 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:31.303197    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz" event={"ID":"2fd6f85b-2fd5-456a-919c-ca59d7e5203f","Type":"ContainerStarted","Data":"9fe9635db8b26b0ce4ba19be416aca98f70219de3353fa56684f4dc2fa1479e5"}
Sep 18 19:38:31.303449 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:31.303430    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz"
Sep 18 19:38:31.303560 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:31.303481    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz"
Sep 18 19:38:31.303560 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:31.303511    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz"
Sep 18 19:38:31.317546 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:31.317488    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47hjk" podStartSLOduration=5.602767806 podStartE2EDuration="27.317467681s" totalImagesPullingTime="21.714699875s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:38:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:28.302785926 +0000 UTC m=+24.813142017" watchObservedRunningTime="2026-09-18 19:38:31.317467681 +0000 UTC m=+27.827823774"
Sep 18 19:38:31.319574 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:31.319547    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz"
Sep 18 19:38:31.319712 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:31.319654    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz"
Sep 18 19:38:31.338860 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:31.338815    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz" podStartSLOduration=7.354378691 podStartE2EDuration="27.338798613s" totalImagesPullingTime="19.984419922s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:38:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:31.338656631 +0000 UTC m=+27.849012746" watchObservedRunningTime="2026-09-18 19:38:31.338798613 +0000 UTC m=+27.849154703"
Sep 18 19:38:32.360327 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:32.360287    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6vdpn"]
Sep 18 19:38:32.361141 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:32.360428    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vdpn"
Sep 18 19:38:32.361141 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:32.360521    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vdpn" podUID="2939ca32-81a7-4e9d-bdc6-ffca73820918"
Sep 18 19:38:32.363847 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:32.363756    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-857bd"]
Sep 18 19:38:32.363910 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:32.363855    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-857bd"
Sep 18 19:38:32.364010 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:32.363946    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-857bd" podUID="1b42bdad-4122-4a0a-978c-d2d5a7bdbd55"
Sep 18 19:38:32.364570 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:32.364545    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xgxbn"]
Sep 18 19:38:32.364670 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:32.364655    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xgxbn"
Sep 18 19:38:32.364799 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:32.364765    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xgxbn" podUID="256a72d8-e079-42fb-9426-c53adc283d1f"
Sep 18 19:38:33.226266 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:33.226223    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-667km"
Sep 18 19:38:33.226421 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:33.226379    2564 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 18 19:38:33.226877 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:33.226859    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-667km"
Sep 18 19:38:33.298662 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:33.298617    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:33.309374 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:33.309337    2564 generic.go:356] "Generic (PLEG): container finished" podID="6df88db0-10f0-429d-b36f-644069df51ed" containerID="c15b74c6dab30727ba1a8cc32673bec218dc1b8aafa428148682f5c0a3688787" exitCode=0
Sep 18 19:38:33.309527 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:33.309404    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-szj4k" event={"ID":"6df88db0-10f0-429d-b36f-644069df51ed","Type":"ContainerDied","Data":"c15b74c6dab30727ba1a8cc32673bec218dc1b8aafa428148682f5c0a3688787"}
Sep 18 19:38:33.380886 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:33.380851    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:33.994709 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:33.994672    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xgxbn"
Sep 18 19:38:33.994904 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:33.994787    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-857bd"
Sep 18 19:38:33.994904 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:33.994804    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vdpn"
Sep 18 19:38:33.994904 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:33.994790    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xgxbn" podUID="256a72d8-e079-42fb-9426-c53adc283d1f"
Sep 18 19:38:33.994904 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:33.994883    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-857bd" podUID="1b42bdad-4122-4a0a-978c-d2d5a7bdbd55"
Sep 18 19:38:33.995224 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:33.994953    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vdpn" podUID="2939ca32-81a7-4e9d-bdc6-ffca73820918"
Sep 18 19:38:35.315714 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:35.315679    2564 generic.go:356] "Generic (PLEG): container finished" podID="6df88db0-10f0-429d-b36f-644069df51ed" containerID="d6b43fabb8bb63ba25e427d13a3e20a61f8da32c7f6c3848dd2a40ed6cac029e" exitCode=0
Sep 18 19:38:35.316169 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:35.315749    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-szj4k" event={"ID":"6df88db0-10f0-429d-b36f-644069df51ed","Type":"ContainerDied","Data":"d6b43fabb8bb63ba25e427d13a3e20a61f8da32c7f6c3848dd2a40ed6cac029e"}
Sep 18 19:38:35.851899 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:35.851861    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:35.905678 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:35.905643    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:35.997392 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:35.997356    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vdpn"
Sep 18 19:38:35.997559 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:35.997408    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xgxbn"
Sep 18 19:38:35.997559 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:35.997522    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-857bd"
Sep 18 19:38:35.997680 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:35.997552    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vdpn" podUID="2939ca32-81a7-4e9d-bdc6-ffca73820918"
Sep 18 19:38:35.997680 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:35.997609    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-857bd" podUID="1b42bdad-4122-4a0a-978c-d2d5a7bdbd55"
Sep 18 19:38:35.997781 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:35.997681    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xgxbn" podUID="256a72d8-e079-42fb-9426-c53adc283d1f"
Sep 18 19:38:36.637054 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:36.637021    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/256a72d8-e079-42fb-9426-c53adc283d1f-metrics-certs\") pod \"network-metrics-daemon-xgxbn\" (UID: \"256a72d8-e079-42fb-9426-c53adc283d1f\") " pod="openshift-multus/network-metrics-daemon-xgxbn"
Sep 18 19:38:36.637533 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:36.637167    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:36.637533 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:36.637245    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/256a72d8-e079-42fb-9426-c53adc283d1f-metrics-certs podName:256a72d8-e079-42fb-9426-c53adc283d1f nodeName:}" failed. No retries permitted until 2026-09-18 19:39:08.637225637 +0000 UTC m=+65.147581725 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/256a72d8-e079-42fb-9426-c53adc283d1f-metrics-certs") pod "network-metrics-daemon-xgxbn" (UID: "256a72d8-e079-42fb-9426-c53adc283d1f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:36.737910 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:36.737866    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2939ca32-81a7-4e9d-bdc6-ffca73820918-original-pull-secret\") pod \"global-pull-secret-syncer-6vdpn\" (UID: \"2939ca32-81a7-4e9d-bdc6-ffca73820918\") " pod="kube-system/global-pull-secret-syncer-6vdpn"
Sep 18 19:38:36.738119 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:36.738031    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:36.738119 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:36.738104    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2939ca32-81a7-4e9d-bdc6-ffca73820918-original-pull-secret podName:2939ca32-81a7-4e9d-bdc6-ffca73820918 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:08.738083055 +0000 UTC m=+65.248439128 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2939ca32-81a7-4e9d-bdc6-ffca73820918-original-pull-secret") pod "global-pull-secret-syncer-6vdpn" (UID: "2939ca32-81a7-4e9d-bdc6-ffca73820918") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:36.838543 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:36.838497    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjgc5\" (UniqueName: \"kubernetes.io/projected/1b42bdad-4122-4a0a-978c-d2d5a7bdbd55-kube-api-access-zjgc5\") pod \"network-check-target-857bd\" (UID: \"1b42bdad-4122-4a0a-978c-d2d5a7bdbd55\") " pod="openshift-network-diagnostics/network-check-target-857bd"
Sep 18 19:38:36.838749 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:36.838697    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:38:36.838749 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:36.838723    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:38:36.838749 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:36.838738    2564 projected.go:196] Error preparing data for projected volume kube-api-access-zjgc5 for pod openshift-network-diagnostics/network-check-target-857bd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:36.838903 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:36.838811    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b42bdad-4122-4a0a-978c-d2d5a7bdbd55-kube-api-access-zjgc5 podName:1b42bdad-4122-4a0a-978c-d2d5a7bdbd55 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:08.838790737 +0000 UTC m=+65.349146827 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-zjgc5" (UniqueName: "kubernetes.io/projected/1b42bdad-4122-4a0a-978c-d2d5a7bdbd55-kube-api-access-zjgc5") pod "network-check-target-857bd" (UID: "1b42bdad-4122-4a0a-978c-d2d5a7bdbd55") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:37.309215 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:37.309176    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:37.366268 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:37.366221    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:37.814698 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:37.814666    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-161.ec2.internal" event="NodeReady"
Sep 18 19:38:37.815107 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:37.814791    2564 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 18 19:38:37.865847 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:37.865760    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-5krlr"]
Sep 18 19:38:37.886541 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:37.886503    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-d8v8m"]
Sep 18 19:38:37.886708 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:37.886593    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5krlr"
Sep 18 19:38:37.888669 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:37.888638    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5f8sq\""
Sep 18 19:38:37.888797 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:37.888680    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 18 19:38:37.888960 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:37.888936    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 18 19:38:37.904492 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:37.904464    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5krlr"]
Sep 18 19:38:37.904660 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:37.904553    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-d8v8m"]
Sep 18 19:38:37.904723 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:37.904661    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-d8v8m"
Sep 18 19:38:37.906838 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:37.906806    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 18 19:38:37.906838 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:37.906820    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 18 19:38:37.907015 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:37.906937    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 18 19:38:37.907073 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:37.907056    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-q4x2f\""
Sep 18 19:38:37.994215 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:37.994172    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-857bd"
Sep 18 19:38:37.994391 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:37.994172    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xgxbn"
Sep 18 19:38:37.994391 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:37.994184    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vdpn"
Sep 18 19:38:37.996880 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:37.996855    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-pwxm5\""
Sep 18 19:38:37.997031 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:37.996887    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 19:38:37.997031 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:37.996906    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 19:38:37.997031 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:37.996915    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 19:38:37.997031 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:37.996906    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8c9m8\""
Sep 18 19:38:37.997308 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:37.997290    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 19:38:38.048392 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:38.048354    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zj6t\" (UniqueName: \"kubernetes.io/projected/218f155a-deaf-4ea6-8234-74b776e05a7b-kube-api-access-7zj6t\") pod \"ingress-canary-d8v8m\" (UID: \"218f155a-deaf-4ea6-8234-74b776e05a7b\") " pod="openshift-ingress-canary/ingress-canary-d8v8m"
Sep 18 19:38:38.048584 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:38.048424    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/218f155a-deaf-4ea6-8234-74b776e05a7b-cert\") pod \"ingress-canary-d8v8m\" (UID: \"218f155a-deaf-4ea6-8234-74b776e05a7b\") " pod="openshift-ingress-canary/ingress-canary-d8v8m"
Sep 18 19:38:38.048584 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:38.048568    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8b6ee93c-e19d-44f2-9ba9-ae133edb7686-config-volume\") pod \"dns-default-5krlr\" (UID: \"8b6ee93c-e19d-44f2-9ba9-ae133edb7686\") " pod="openshift-dns/dns-default-5krlr"
Sep 18 19:38:38.048693 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:38.048603    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b6ee93c-e19d-44f2-9ba9-ae133edb7686-tmp-dir\") pod \"dns-default-5krlr\" (UID: \"8b6ee93c-e19d-44f2-9ba9-ae133edb7686\") " pod="openshift-dns/dns-default-5krlr"
Sep 18 19:38:38.048740 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:38.048693    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfm65\" (UniqueName: \"kubernetes.io/projected/8b6ee93c-e19d-44f2-9ba9-ae133edb7686-kube-api-access-gfm65\") pod \"dns-default-5krlr\" (UID: \"8b6ee93c-e19d-44f2-9ba9-ae133edb7686\") " pod="openshift-dns/dns-default-5krlr"
Sep 18 19:38:38.048740 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:38.048716    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8b6ee93c-e19d-44f2-9ba9-ae133edb7686-metrics-tls\") pod \"dns-default-5krlr\" (UID: \"8b6ee93c-e19d-44f2-9ba9-ae133edb7686\") " pod="openshift-dns/dns-default-5krlr"
Sep 18 19:38:38.149619 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:38.149582    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8b6ee93c-e19d-44f2-9ba9-ae133edb7686-metrics-tls\") pod \"dns-default-5krlr\" (UID: \"8b6ee93c-e19d-44f2-9ba9-ae133edb7686\") " pod="openshift-dns/dns-default-5krlr"
Sep 18 19:38:38.149805 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:38.149629    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zj6t\" (UniqueName: \"kubernetes.io/projected/218f155a-deaf-4ea6-8234-74b776e05a7b-kube-api-access-7zj6t\") pod \"ingress-canary-d8v8m\" (UID: \"218f155a-deaf-4ea6-8234-74b776e05a7b\") " pod="openshift-ingress-canary/ingress-canary-d8v8m"
Sep 18 19:38:38.149805 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:38.149672    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/218f155a-deaf-4ea6-8234-74b776e05a7b-cert\") pod \"ingress-canary-d8v8m\" (UID: \"218f155a-deaf-4ea6-8234-74b776e05a7b\") " pod="openshift-ingress-canary/ingress-canary-d8v8m"
Sep 18 19:38:38.149805 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:38.149718    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8b6ee93c-e19d-44f2-9ba9-ae133edb7686-config-volume\") pod \"dns-default-5krlr\" (UID: \"8b6ee93c-e19d-44f2-9ba9-ae133edb7686\") " pod="openshift-dns/dns-default-5krlr"
Sep 18 19:38:38.149805 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:38.149740    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b6ee93c-e19d-44f2-9ba9-ae133edb7686-tmp-dir\") pod \"dns-default-5krlr\" (UID: \"8b6ee93c-e19d-44f2-9ba9-ae133edb7686\") " pod="openshift-dns/dns-default-5krlr"
Sep 18 19:38:38.149805 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:38.149752    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:38:38.150016 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:38.149810    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfm65\" (UniqueName: \"kubernetes.io/projected/8b6ee93c-e19d-44f2-9ba9-ae133edb7686-kube-api-access-gfm65\") pod \"dns-default-5krlr\" (UID: \"8b6ee93c-e19d-44f2-9ba9-ae133edb7686\") " pod="openshift-dns/dns-default-5krlr"
Sep 18 19:38:38.150016 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:38.149825    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b6ee93c-e19d-44f2-9ba9-ae133edb7686-metrics-tls podName:8b6ee93c-e19d-44f2-9ba9-ae133edb7686 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:38.649808427 +0000 UTC m=+35.160164515 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8b6ee93c-e19d-44f2-9ba9-ae133edb7686-metrics-tls") pod "dns-default-5krlr" (UID: "8b6ee93c-e19d-44f2-9ba9-ae133edb7686") : secret "dns-default-metrics-tls" not found
Sep 18 19:38:38.150214 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:38.150194    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b6ee93c-e19d-44f2-9ba9-ae133edb7686-tmp-dir\") pod \"dns-default-5krlr\" (UID: \"8b6ee93c-e19d-44f2-9ba9-ae133edb7686\") " pod="openshift-dns/dns-default-5krlr"
Sep 18 19:38:38.150378 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:38.150362    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8b6ee93c-e19d-44f2-9ba9-ae133edb7686-config-volume\") pod \"dns-default-5krlr\" (UID: \"8b6ee93c-e19d-44f2-9ba9-ae133edb7686\") " pod="openshift-dns/dns-default-5krlr"
Sep 18 19:38:38.154709 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:38.154684    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/218f155a-deaf-4ea6-8234-74b776e05a7b-cert\") pod \"ingress-canary-d8v8m\" (UID: \"218f155a-deaf-4ea6-8234-74b776e05a7b\") " pod="openshift-ingress-canary/ingress-canary-d8v8m"
Sep 18 19:38:38.158745 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:38.158677    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfm65\" (UniqueName: \"kubernetes.io/projected/8b6ee93c-e19d-44f2-9ba9-ae133edb7686-kube-api-access-gfm65\") pod \"dns-default-5krlr\" (UID: \"8b6ee93c-e19d-44f2-9ba9-ae133edb7686\") " pod="openshift-dns/dns-default-5krlr"
Sep 18 19:38:38.159477 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:38.159443    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zj6t\" (UniqueName: \"kubernetes.io/projected/218f155a-deaf-4ea6-8234-74b776e05a7b-kube-api-access-7zj6t\") pod \"ingress-canary-d8v8m\" (UID: \"218f155a-deaf-4ea6-8234-74b776e05a7b\") " pod="openshift-ingress-canary/ingress-canary-d8v8m"
Sep 18 19:38:38.214430 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:38.214391    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-d8v8m"
Sep 18 19:38:38.653723 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:38.653683    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8b6ee93c-e19d-44f2-9ba9-ae133edb7686-metrics-tls\") pod \"dns-default-5krlr\" (UID: \"8b6ee93c-e19d-44f2-9ba9-ae133edb7686\") " pod="openshift-dns/dns-default-5krlr"
Sep 18 19:38:38.653904 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:38.653810    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:38:38.653904 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:38.653876    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b6ee93c-e19d-44f2-9ba9-ae133edb7686-metrics-tls podName:8b6ee93c-e19d-44f2-9ba9-ae133edb7686 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:39.653861363 +0000 UTC m=+36.164217455 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8b6ee93c-e19d-44f2-9ba9-ae133edb7686-metrics-tls") pod "dns-default-5krlr" (UID: "8b6ee93c-e19d-44f2-9ba9-ae133edb7686") : secret "dns-default-metrics-tls" not found
Sep 18 19:38:39.662565 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:39.662356    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8b6ee93c-e19d-44f2-9ba9-ae133edb7686-metrics-tls\") pod \"dns-default-5krlr\" (UID: \"8b6ee93c-e19d-44f2-9ba9-ae133edb7686\") " pod="openshift-dns/dns-default-5krlr"
Sep 18 19:38:39.662565 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:39.662546    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:38:39.663071 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:39.662653    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b6ee93c-e19d-44f2-9ba9-ae133edb7686-metrics-tls podName:8b6ee93c-e19d-44f2-9ba9-ae133edb7686 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:41.6626303 +0000 UTC m=+38.172986372 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8b6ee93c-e19d-44f2-9ba9-ae133edb7686-metrics-tls") pod "dns-default-5krlr" (UID: "8b6ee93c-e19d-44f2-9ba9-ae133edb7686") : secret "dns-default-metrics-tls" not found
Sep 18 19:38:39.717130 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:39.717100    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-d8v8m"]
Sep 18 19:38:39.753860 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:39.753830    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod218f155a_deaf_4ea6_8234_74b776e05a7b.slice/crio-1ebad0ee41bd831ec94f9e5e0e48d0f08ba31d217cd4e2b061b8eb0ae71909da WatchSource:0}: Error finding container 1ebad0ee41bd831ec94f9e5e0e48d0f08ba31d217cd4e2b061b8eb0ae71909da: Status 404 returned error can't find the container with id 1ebad0ee41bd831ec94f9e5e0e48d0f08ba31d217cd4e2b061b8eb0ae71909da
Sep 18 19:38:40.330824 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:40.330785    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-d8v8m" event={"ID":"218f155a-deaf-4ea6-8234-74b776e05a7b","Type":"ContainerStarted","Data":"1ebad0ee41bd831ec94f9e5e0e48d0f08ba31d217cd4e2b061b8eb0ae71909da"}
Sep 18 19:38:40.334221 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:40.334187    2564 generic.go:356] "Generic (PLEG): container finished" podID="6df88db0-10f0-429d-b36f-644069df51ed" containerID="2c5b0781c9af362d37d24c53b95a2b304456512b6b4c2cf1ebdab3d2b220c7b8" exitCode=0
Sep 18 19:38:40.334380 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:40.334248    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-szj4k" event={"ID":"6df88db0-10f0-429d-b36f-644069df51ed","Type":"ContainerDied","Data":"2c5b0781c9af362d37d24c53b95a2b304456512b6b4c2cf1ebdab3d2b220c7b8"}
Sep 18 19:38:41.340851 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:41.340810    2564 generic.go:356] "Generic (PLEG): container finished" podID="6df88db0-10f0-429d-b36f-644069df51ed" containerID="ea418d5b9ee73028f7c953feaff14844e36e87c09da498a88a6c86e1ac5e4189" exitCode=0
Sep 18 19:38:41.341610 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:41.340866    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-szj4k" event={"ID":"6df88db0-10f0-429d-b36f-644069df51ed","Type":"ContainerDied","Data":"ea418d5b9ee73028f7c953feaff14844e36e87c09da498a88a6c86e1ac5e4189"}
Sep 18 19:38:41.677545 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:41.677449    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8b6ee93c-e19d-44f2-9ba9-ae133edb7686-metrics-tls\") pod \"dns-default-5krlr\" (UID: \"8b6ee93c-e19d-44f2-9ba9-ae133edb7686\") " pod="openshift-dns/dns-default-5krlr"
Sep 18 19:38:41.677718 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:41.677625    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:38:41.677772 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:41.677732    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b6ee93c-e19d-44f2-9ba9-ae133edb7686-metrics-tls podName:8b6ee93c-e19d-44f2-9ba9-ae133edb7686 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:45.677714662 +0000 UTC m=+42.188070731 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8b6ee93c-e19d-44f2-9ba9-ae133edb7686-metrics-tls") pod "dns-default-5krlr" (UID: "8b6ee93c-e19d-44f2-9ba9-ae133edb7686") : secret "dns-default-metrics-tls" not found
Sep 18 19:38:42.234386 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:42.234151    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:42.284288 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:42.284252    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:42.345239 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:42.345205    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-d8v8m" event={"ID":"218f155a-deaf-4ea6-8234-74b776e05a7b","Type":"ContainerStarted","Data":"fa883bbfecfe8737f17b04b7f6adca45d50ef7761cfdc26004d308a51301f27f"}
Sep 18 19:38:42.348745 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:42.348710    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-szj4k" event={"ID":"6df88db0-10f0-429d-b36f-644069df51ed","Type":"ContainerStarted","Data":"f8bcce5e3b0a922218fa896991840ed703bf2ad73413d2588152a77aa449a506"}
Sep 18 19:38:42.359449 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:42.359406    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-d8v8m" podStartSLOduration=3.298073605 podStartE2EDuration="5.359390156s" totalImagesPullingTime="2.061316551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:38:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:42.358637027 +0000 UTC m=+38.868993142" watchObservedRunningTime="2026-09-18 19:38:42.359390156 +0000 UTC m=+38.869746246"
Sep 18 19:38:42.380343 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:42.380291    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-szj4k" podStartSLOduration=8.213435304 podStartE2EDuration="38.380273302s" totalImagesPullingTime="30.166837998s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:38:04 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:42.379787025 +0000 UTC m=+38.890143136" watchObservedRunningTime="2026-09-18 19:38:42.380273302 +0000 UTC m=+38.890629393"
Sep 18 19:38:42.500775 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:42.500673    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:42.600707 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:42.600669    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:44.710412 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:44.710375    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:44.812603 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:44.812556    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:45.703941 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:45.703908    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8b6ee93c-e19d-44f2-9ba9-ae133edb7686-metrics-tls\") pod \"dns-default-5krlr\" (UID: \"8b6ee93c-e19d-44f2-9ba9-ae133edb7686\") " pod="openshift-dns/dns-default-5krlr"
Sep 18 19:38:45.704145 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:45.704046    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:38:45.704145 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:45.704104    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b6ee93c-e19d-44f2-9ba9-ae133edb7686-metrics-tls podName:8b6ee93c-e19d-44f2-9ba9-ae133edb7686 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:53.704089027 +0000 UTC m=+50.214445095 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8b6ee93c-e19d-44f2-9ba9-ae133edb7686-metrics-tls") pod "dns-default-5krlr" (UID: "8b6ee93c-e19d-44f2-9ba9-ae133edb7686") : secret "dns-default-metrics-tls" not found
Sep 18 19:38:49.320753 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:49.320720    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:49.406615 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:49.406583    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:49.465275 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:49.465246    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:50.329964 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:50.329929    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:50.430697 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:50.430601    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:53.762865 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:53.762826    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8b6ee93c-e19d-44f2-9ba9-ae133edb7686-metrics-tls\") pod \"dns-default-5krlr\" (UID: \"8b6ee93c-e19d-44f2-9ba9-ae133edb7686\") " pod="openshift-dns/dns-default-5krlr"
Sep 18 19:38:53.763396 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:53.762955    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:38:53.763396 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:53.763042    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b6ee93c-e19d-44f2-9ba9-ae133edb7686-metrics-tls podName:8b6ee93c-e19d-44f2-9ba9-ae133edb7686 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:09.763027313 +0000 UTC m=+66.273383382 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8b6ee93c-e19d-44f2-9ba9-ae133edb7686-metrics-tls") pod "dns-default-5krlr" (UID: "8b6ee93c-e19d-44f2-9ba9-ae133edb7686") : secret "dns-default-metrics-tls" not found
Sep 18 19:38:56.511207 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.511121    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-lshmd"]
Sep 18 19:38:56.533261 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.533232    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-lshmd"]
Sep 18 19:38:56.533432 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.533384    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-lshmd"
Sep 18 19:38:56.535414 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.535392    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 18 19:38:56.535555 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.535396    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-2bngb\""
Sep 18 19:38:56.536072 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.536050    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 18 19:38:56.536072 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.536064    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 18 19:38:56.536283 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.536103    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 18 19:38:56.632592 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.632554    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-756fcb9d6-47xpw"]
Sep 18 19:38:56.658502 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.658473    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-756fcb9d6-47xpw"]
Sep 18 19:38:56.658659 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.658572    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-756fcb9d6-47xpw"
Sep 18 19:38:56.660512 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.660479    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 18 19:38:56.660687 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.660498    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 18 19:38:56.660687 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.660571    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 18 19:38:56.660851 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.660822    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 18 19:38:56.660851 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.660845    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 18 19:38:56.661034 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.660938    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 18 19:38:56.661275 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.661255    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-9lg52\""
Sep 18 19:38:56.684610 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.684577    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9b05425f-3a0e-40b6-b1af-4f3098050245-serving-cert\") pod \"service-ca-operator-57cdc4b768-lshmd\" (UID: \"9b05425f-3a0e-40b6-b1af-4f3098050245\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-lshmd"
Sep 18 19:38:56.684776 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.684626    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnvzh\" (UniqueName: \"kubernetes.io/projected/9b05425f-3a0e-40b6-b1af-4f3098050245-kube-api-access-vnvzh\") pod \"service-ca-operator-57cdc4b768-lshmd\" (UID: \"9b05425f-3a0e-40b6-b1af-4f3098050245\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-lshmd"
Sep 18 19:38:56.684776 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.684715    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9b05425f-3a0e-40b6-b1af-4f3098050245-config\") pod \"service-ca-operator-57cdc4b768-lshmd\" (UID: \"9b05425f-3a0e-40b6-b1af-4f3098050245\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-lshmd"
Sep 18 19:38:56.719893 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.719732    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-58vl4"]
Sep 18 19:38:56.740405 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.740373    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-w4mzh"]
Sep 18 19:38:56.740560 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.740509    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-58vl4"
Sep 18 19:38:56.743547 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.743515    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5f57555bf7-2b66f"]
Sep 18 19:38:56.743826 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.743577    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 18 19:38:56.743953 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.743594    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 18 19:38:56.744090 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.743636    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 18 19:38:56.744090 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.743671    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-w4mzh"
Sep 18 19:38:56.744207 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.743704    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 18 19:38:56.745605 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.745554    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-q27dg\""
Sep 18 19:38:56.746038 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.746016    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-58vl4"]
Sep 18 19:38:56.746142 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.746055    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5f57555bf7-2b66f"]
Sep 18 19:38:56.746142 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.746074    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-w4mzh"]
Sep 18 19:38:56.746300 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.746183    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f57555bf7-2b66f"
Sep 18 19:38:56.747762 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.747723    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 18 19:38:56.747878 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.747760    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 18 19:38:56.747878 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.747771    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 18 19:38:56.748116 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.748097    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 18 19:38:56.748189 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.748104    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-x4hwn\""
Sep 18 19:38:56.748189 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.748137    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 18 19:38:56.748319 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.748301    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-ntrxj\""
Sep 18 19:38:56.748734 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.748716    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 18 19:38:56.749760 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.749667    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 18 19:38:56.753595 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.753574    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 18 19:38:56.778663 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.778630    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:56.785583 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.785549    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9b05425f-3a0e-40b6-b1af-4f3098050245-serving-cert\") pod \"service-ca-operator-57cdc4b768-lshmd\" (UID: \"9b05425f-3a0e-40b6-b1af-4f3098050245\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-lshmd"
Sep 18 19:38:56.785686 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.785595    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpgjq\" (UniqueName: \"kubernetes.io/projected/20027a0c-0095-4418-aaa4-d0a6f273ef85-kube-api-access-xpgjq\") pod \"router-default-756fcb9d6-47xpw\" (UID: \"20027a0c-0095-4418-aaa4-d0a6f273ef85\") " pod="openshift-ingress/router-default-756fcb9d6-47xpw"
Sep 18 19:38:56.785686 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.785640    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20027a0c-0095-4418-aaa4-d0a6f273ef85-service-ca-bundle\") pod \"router-default-756fcb9d6-47xpw\" (UID: \"20027a0c-0095-4418-aaa4-d0a6f273ef85\") " pod="openshift-ingress/router-default-756fcb9d6-47xpw"
Sep 18 19:38:56.785686 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.785668    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnvzh\" (UniqueName: \"kubernetes.io/projected/9b05425f-3a0e-40b6-b1af-4f3098050245-kube-api-access-vnvzh\") pod \"service-ca-operator-57cdc4b768-lshmd\" (UID: \"9b05425f-3a0e-40b6-b1af-4f3098050245\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-lshmd"
Sep 18 19:38:56.785796 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.785732    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/20027a0c-0095-4418-aaa4-d0a6f273ef85-stats-auth\") pod \"router-default-756fcb9d6-47xpw\" (UID: \"20027a0c-0095-4418-aaa4-d0a6f273ef85\") " pod="openshift-ingress/router-default-756fcb9d6-47xpw"
Sep 18 19:38:56.785796 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.785765    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20027a0c-0095-4418-aaa4-d0a6f273ef85-metrics-certs\") pod \"router-default-756fcb9d6-47xpw\" (UID: \"20027a0c-0095-4418-aaa4-d0a6f273ef85\") " pod="openshift-ingress/router-default-756fcb9d6-47xpw"
Sep 18 19:38:56.785869 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.785809    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/20027a0c-0095-4418-aaa4-d0a6f273ef85-default-certificate\") pod \"router-default-756fcb9d6-47xpw\" (UID: \"20027a0c-0095-4418-aaa4-d0a6f273ef85\") " pod="openshift-ingress/router-default-756fcb9d6-47xpw"
Sep 18 19:38:56.785869 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.785844    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9b05425f-3a0e-40b6-b1af-4f3098050245-config\") pod \"service-ca-operator-57cdc4b768-lshmd\" (UID: \"9b05425f-3a0e-40b6-b1af-4f3098050245\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-lshmd"
Sep 18 19:38:56.786362 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.786344    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9b05425f-3a0e-40b6-b1af-4f3098050245-config\") pod \"service-ca-operator-57cdc4b768-lshmd\" (UID: \"9b05425f-3a0e-40b6-b1af-4f3098050245\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-lshmd"
Sep 18 19:38:56.788767 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.788745    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9b05425f-3a0e-40b6-b1af-4f3098050245-serving-cert\") pod \"service-ca-operator-57cdc4b768-lshmd\" (UID: \"9b05425f-3a0e-40b6-b1af-4f3098050245\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-lshmd"
Sep 18 19:38:56.794791 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.794760    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnvzh\" (UniqueName: \"kubernetes.io/projected/9b05425f-3a0e-40b6-b1af-4f3098050245-kube-api-access-vnvzh\") pod \"service-ca-operator-57cdc4b768-lshmd\" (UID: \"9b05425f-3a0e-40b6-b1af-4f3098050245\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-lshmd"
Sep 18 19:38:56.833117 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.833084    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:56.842900 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.842868    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-lshmd"
Sep 18 19:38:56.886544 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.886507    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpgjq\" (UniqueName: \"kubernetes.io/projected/20027a0c-0095-4418-aaa4-d0a6f273ef85-kube-api-access-xpgjq\") pod \"router-default-756fcb9d6-47xpw\" (UID: \"20027a0c-0095-4418-aaa4-d0a6f273ef85\") " pod="openshift-ingress/router-default-756fcb9d6-47xpw"
Sep 18 19:38:56.886544 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.886547    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6nz8\" (UniqueName: \"kubernetes.io/projected/945a444e-9381-4e2f-b4c1-64be0ec88e6c-kube-api-access-w6nz8\") pod \"console-operator-fd4bb46c-58vl4\" (UID: \"945a444e-9381-4e2f-b4c1-64be0ec88e6c\") " pod="openshift-console-operator/console-operator-fd4bb46c-58vl4"
Sep 18 19:38:56.886737 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.886564    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7672fd63-163a-481c-aaf8-a1447da0fbf0-trusted-ca\") pod \"image-registry-5f57555bf7-2b66f\" (UID: \"7672fd63-163a-481c-aaf8-a1447da0fbf0\") " pod="openshift-image-registry/image-registry-5f57555bf7-2b66f"
Sep 18 19:38:56.886737 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.886631    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/945a444e-9381-4e2f-b4c1-64be0ec88e6c-serving-cert\") pod \"console-operator-fd4bb46c-58vl4\" (UID: \"945a444e-9381-4e2f-b4c1-64be0ec88e6c\") " pod="openshift-console-operator/console-operator-fd4bb46c-58vl4"
Sep 18 19:38:56.886737 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.886676    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/18f89927-0873-43f7-b237-ec56aaacb7c0-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-w4mzh\" (UID: \"18f89927-0873-43f7-b237-ec56aaacb7c0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-w4mzh"
Sep 18 19:38:56.886835 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.886751    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20027a0c-0095-4418-aaa4-d0a6f273ef85-service-ca-bundle\") pod \"router-default-756fcb9d6-47xpw\" (UID: \"20027a0c-0095-4418-aaa4-d0a6f273ef85\") " pod="openshift-ingress/router-default-756fcb9d6-47xpw"
Sep 18 19:38:56.886835 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.886778    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzzl2\" (UniqueName: \"kubernetes.io/projected/18f89927-0873-43f7-b237-ec56aaacb7c0-kube-api-access-nzzl2\") pod \"cluster-samples-operator-685d7d68b5-w4mzh\" (UID: \"18f89927-0873-43f7-b237-ec56aaacb7c0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-w4mzh"
Sep 18 19:38:56.886835 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.886805    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7672fd63-163a-481c-aaf8-a1447da0fbf0-registry-certificates\") pod \"image-registry-5f57555bf7-2b66f\" (UID: \"7672fd63-163a-481c-aaf8-a1447da0fbf0\") " pod="openshift-image-registry/image-registry-5f57555bf7-2b66f"
Sep 18 19:38:56.886949 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.886833    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7672fd63-163a-481c-aaf8-a1447da0fbf0-installation-pull-secrets\") pod \"image-registry-5f57555bf7-2b66f\" (UID: \"7672fd63-163a-481c-aaf8-a1447da0fbf0\") " pod="openshift-image-registry/image-registry-5f57555bf7-2b66f"
Sep 18 19:38:56.886949 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.886872    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/945a444e-9381-4e2f-b4c1-64be0ec88e6c-trusted-ca\") pod \"console-operator-fd4bb46c-58vl4\" (UID: \"945a444e-9381-4e2f-b4c1-64be0ec88e6c\") " pod="openshift-console-operator/console-operator-fd4bb46c-58vl4"
Sep 18 19:38:56.886949 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.886897    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7672fd63-163a-481c-aaf8-a1447da0fbf0-ca-trust-extracted\") pod \"image-registry-5f57555bf7-2b66f\" (UID: \"7672fd63-163a-481c-aaf8-a1447da0fbf0\") " pod="openshift-image-registry/image-registry-5f57555bf7-2b66f"
Sep 18 19:38:56.886949 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:56.886914    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/20027a0c-0095-4418-aaa4-d0a6f273ef85-service-ca-bundle podName:20027a0c-0095-4418-aaa4-d0a6f273ef85 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:57.386892231 +0000 UTC m=+53.897248312 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/20027a0c-0095-4418-aaa4-d0a6f273ef85-service-ca-bundle") pod "router-default-756fcb9d6-47xpw" (UID: "20027a0c-0095-4418-aaa4-d0a6f273ef85") : configmap references non-existent config key: service-ca.crt
Sep 18 19:38:56.887094 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.887013    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/945a444e-9381-4e2f-b4c1-64be0ec88e6c-config\") pod \"console-operator-fd4bb46c-58vl4\" (UID: \"945a444e-9381-4e2f-b4c1-64be0ec88e6c\") " pod="openshift-console-operator/console-operator-fd4bb46c-58vl4"
Sep 18 19:38:56.887094 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.887055    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/20027a0c-0095-4418-aaa4-d0a6f273ef85-stats-auth\") pod \"router-default-756fcb9d6-47xpw\" (UID: \"20027a0c-0095-4418-aaa4-d0a6f273ef85\") " pod="openshift-ingress/router-default-756fcb9d6-47xpw"
Sep 18 19:38:56.887094 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.887090    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20027a0c-0095-4418-aaa4-d0a6f273ef85-metrics-certs\") pod \"router-default-756fcb9d6-47xpw\" (UID: \"20027a0c-0095-4418-aaa4-d0a6f273ef85\") " pod="openshift-ingress/router-default-756fcb9d6-47xpw"
Sep 18 19:38:56.887188 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.887109    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7672fd63-163a-481c-aaf8-a1447da0fbf0-image-registry-private-configuration\") pod \"image-registry-5f57555bf7-2b66f\" (UID: \"7672fd63-163a-481c-aaf8-a1447da0fbf0\") " pod="openshift-image-registry/image-registry-5f57555bf7-2b66f"
Sep 18 19:38:56.887188 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.887133    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qg4x\" (UniqueName: \"kubernetes.io/projected/7672fd63-163a-481c-aaf8-a1447da0fbf0-kube-api-access-5qg4x\") pod \"image-registry-5f57555bf7-2b66f\" (UID: \"7672fd63-163a-481c-aaf8-a1447da0fbf0\") " pod="openshift-image-registry/image-registry-5f57555bf7-2b66f"
Sep 18 19:38:56.887188 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.887177    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/20027a0c-0095-4418-aaa4-d0a6f273ef85-default-certificate\") pod \"router-default-756fcb9d6-47xpw\" (UID: \"20027a0c-0095-4418-aaa4-d0a6f273ef85\") " pod="openshift-ingress/router-default-756fcb9d6-47xpw"
Sep 18 19:38:56.887414 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:56.887252    2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 19:38:56.887414 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.887294    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7672fd63-163a-481c-aaf8-a1447da0fbf0-bound-sa-token\") pod \"image-registry-5f57555bf7-2b66f\" (UID: \"7672fd63-163a-481c-aaf8-a1447da0fbf0\") " pod="openshift-image-registry/image-registry-5f57555bf7-2b66f"
Sep 18 19:38:56.887414 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:56.887327    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20027a0c-0095-4418-aaa4-d0a6f273ef85-metrics-certs podName:20027a0c-0095-4418-aaa4-d0a6f273ef85 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:57.387309115 +0000 UTC m=+53.897665198 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/20027a0c-0095-4418-aaa4-d0a6f273ef85-metrics-certs") pod "router-default-756fcb9d6-47xpw" (UID: "20027a0c-0095-4418-aaa4-d0a6f273ef85") : secret "router-metrics-certs-default" not found
Sep 18 19:38:56.887414 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.887368    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7672fd63-163a-481c-aaf8-a1447da0fbf0-registry-tls\") pod \"image-registry-5f57555bf7-2b66f\" (UID: \"7672fd63-163a-481c-aaf8-a1447da0fbf0\") " pod="openshift-image-registry/image-registry-5f57555bf7-2b66f"
Sep 18 19:38:56.889805 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.889781    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/20027a0c-0095-4418-aaa4-d0a6f273ef85-stats-auth\") pod \"router-default-756fcb9d6-47xpw\" (UID: \"20027a0c-0095-4418-aaa4-d0a6f273ef85\") " pod="openshift-ingress/router-default-756fcb9d6-47xpw"
Sep 18 19:38:56.890390 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.890367    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/20027a0c-0095-4418-aaa4-d0a6f273ef85-default-certificate\") pod \"router-default-756fcb9d6-47xpw\" (UID: \"20027a0c-0095-4418-aaa4-d0a6f273ef85\") " pod="openshift-ingress/router-default-756fcb9d6-47xpw"
Sep 18 19:38:56.895703 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.895675    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpgjq\" (UniqueName: \"kubernetes.io/projected/20027a0c-0095-4418-aaa4-d0a6f273ef85-kube-api-access-xpgjq\") pod \"router-default-756fcb9d6-47xpw\" (UID: \"20027a0c-0095-4418-aaa4-d0a6f273ef85\") " pod="openshift-ingress/router-default-756fcb9d6-47xpw"
Sep 18 19:38:56.988213 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.988173    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6nz8\" (UniqueName: \"kubernetes.io/projected/945a444e-9381-4e2f-b4c1-64be0ec88e6c-kube-api-access-w6nz8\") pod \"console-operator-fd4bb46c-58vl4\" (UID: \"945a444e-9381-4e2f-b4c1-64be0ec88e6c\") " pod="openshift-console-operator/console-operator-fd4bb46c-58vl4"
Sep 18 19:38:56.988213 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.988215    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7672fd63-163a-481c-aaf8-a1447da0fbf0-trusted-ca\") pod \"image-registry-5f57555bf7-2b66f\" (UID: \"7672fd63-163a-481c-aaf8-a1447da0fbf0\") " pod="openshift-image-registry/image-registry-5f57555bf7-2b66f"
Sep 18 19:38:56.988557 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.988249    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/945a444e-9381-4e2f-b4c1-64be0ec88e6c-serving-cert\") pod \"console-operator-fd4bb46c-58vl4\" (UID: \"945a444e-9381-4e2f-b4c1-64be0ec88e6c\") " pod="openshift-console-operator/console-operator-fd4bb46c-58vl4"
Sep 18 19:38:56.988557 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.988268    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/18f89927-0873-43f7-b237-ec56aaacb7c0-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-w4mzh\" (UID: \"18f89927-0873-43f7-b237-ec56aaacb7c0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-w4mzh"
Sep 18 19:38:56.988557 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.988303    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzzl2\" (UniqueName: \"kubernetes.io/projected/18f89927-0873-43f7-b237-ec56aaacb7c0-kube-api-access-nzzl2\") pod \"cluster-samples-operator-685d7d68b5-w4mzh\" (UID: \"18f89927-0873-43f7-b237-ec56aaacb7c0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-w4mzh"
Sep 18 19:38:56.988557 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.988327    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7672fd63-163a-481c-aaf8-a1447da0fbf0-registry-certificates\") pod \"image-registry-5f57555bf7-2b66f\" (UID: \"7672fd63-163a-481c-aaf8-a1447da0fbf0\") " pod="openshift-image-registry/image-registry-5f57555bf7-2b66f"
Sep 18 19:38:56.988557 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.988355    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7672fd63-163a-481c-aaf8-a1447da0fbf0-installation-pull-secrets\") pod \"image-registry-5f57555bf7-2b66f\" (UID: \"7672fd63-163a-481c-aaf8-a1447da0fbf0\") " pod="openshift-image-registry/image-registry-5f57555bf7-2b66f"
Sep 18 19:38:56.988557 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:56.988385    2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 19:38:56.988557 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.988395    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/945a444e-9381-4e2f-b4c1-64be0ec88e6c-trusted-ca\") pod \"console-operator-fd4bb46c-58vl4\" (UID: \"945a444e-9381-4e2f-b4c1-64be0ec88e6c\") " pod="openshift-console-operator/console-operator-fd4bb46c-58vl4"
Sep 18 19:38:56.988557 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.988418    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7672fd63-163a-481c-aaf8-a1447da0fbf0-ca-trust-extracted\") pod \"image-registry-5f57555bf7-2b66f\" (UID: \"7672fd63-163a-481c-aaf8-a1447da0fbf0\") " pod="openshift-image-registry/image-registry-5f57555bf7-2b66f"
Sep 18 19:38:56.988557 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:56.988455    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18f89927-0873-43f7-b237-ec56aaacb7c0-samples-operator-tls podName:18f89927-0873-43f7-b237-ec56aaacb7c0 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:57.488434548 +0000 UTC m=+53.998790634 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/18f89927-0873-43f7-b237-ec56aaacb7c0-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-w4mzh" (UID: "18f89927-0873-43f7-b237-ec56aaacb7c0") : secret "samples-operator-tls" not found
Sep 18 19:38:56.988557 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.988479    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/945a444e-9381-4e2f-b4c1-64be0ec88e6c-config\") pod \"console-operator-fd4bb46c-58vl4\" (UID: \"945a444e-9381-4e2f-b4c1-64be0ec88e6c\") " pod="openshift-console-operator/console-operator-fd4bb46c-58vl4"
Sep 18 19:38:56.988557 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.988548    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7672fd63-163a-481c-aaf8-a1447da0fbf0-image-registry-private-configuration\") pod \"image-registry-5f57555bf7-2b66f\" (UID: \"7672fd63-163a-481c-aaf8-a1447da0fbf0\") " pod="openshift-image-registry/image-registry-5f57555bf7-2b66f"
Sep 18 19:38:56.989167 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.988574    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qg4x\" (UniqueName: \"kubernetes.io/projected/7672fd63-163a-481c-aaf8-a1447da0fbf0-kube-api-access-5qg4x\") pod \"image-registry-5f57555bf7-2b66f\" (UID: \"7672fd63-163a-481c-aaf8-a1447da0fbf0\") " pod="openshift-image-registry/image-registry-5f57555bf7-2b66f"
Sep 18 19:38:56.989167 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.988616    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7672fd63-163a-481c-aaf8-a1447da0fbf0-bound-sa-token\") pod \"image-registry-5f57555bf7-2b66f\" (UID: \"7672fd63-163a-481c-aaf8-a1447da0fbf0\") " pod="openshift-image-registry/image-registry-5f57555bf7-2b66f"
Sep 18 19:38:56.989167 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.988644    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7672fd63-163a-481c-aaf8-a1447da0fbf0-registry-tls\") pod \"image-registry-5f57555bf7-2b66f\" (UID: \"7672fd63-163a-481c-aaf8-a1447da0fbf0\") " pod="openshift-image-registry/image-registry-5f57555bf7-2b66f"
Sep 18 19:38:56.989167 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.988780    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7672fd63-163a-481c-aaf8-a1447da0fbf0-ca-trust-extracted\") pod \"image-registry-5f57555bf7-2b66f\" (UID: \"7672fd63-163a-481c-aaf8-a1447da0fbf0\") " pod="openshift-image-registry/image-registry-5f57555bf7-2b66f"
Sep 18 19:38:56.989167 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:56.988800    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 19:38:56.989167 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:56.988813    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f57555bf7-2b66f: secret "image-registry-tls" not found
Sep 18 19:38:56.989167 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:56.988858    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7672fd63-163a-481c-aaf8-a1447da0fbf0-registry-tls podName:7672fd63-163a-481c-aaf8-a1447da0fbf0 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:57.48884323 +0000 UTC m=+53.999199313 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7672fd63-163a-481c-aaf8-a1447da0fbf0-registry-tls") pod "image-registry-5f57555bf7-2b66f" (UID: "7672fd63-163a-481c-aaf8-a1447da0fbf0") : secret "image-registry-tls" not found
Sep 18 19:38:56.989465 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.989341    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7672fd63-163a-481c-aaf8-a1447da0fbf0-trusted-ca\") pod \"image-registry-5f57555bf7-2b66f\" (UID: \"7672fd63-163a-481c-aaf8-a1447da0fbf0\") " pod="openshift-image-registry/image-registry-5f57555bf7-2b66f"
Sep 18 19:38:56.989465 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.989426    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/945a444e-9381-4e2f-b4c1-64be0ec88e6c-config\") pod \"console-operator-fd4bb46c-58vl4\" (UID: \"945a444e-9381-4e2f-b4c1-64be0ec88e6c\") " pod="openshift-console-operator/console-operator-fd4bb46c-58vl4"
Sep 18 19:38:56.989768 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.989747    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/945a444e-9381-4e2f-b4c1-64be0ec88e6c-trusted-ca\") pod \"console-operator-fd4bb46c-58vl4\" (UID: \"945a444e-9381-4e2f-b4c1-64be0ec88e6c\") " pod="openshift-console-operator/console-operator-fd4bb46c-58vl4"
Sep 18 19:38:56.989805 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.989791    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7672fd63-163a-481c-aaf8-a1447da0fbf0-registry-certificates\") pod \"image-registry-5f57555bf7-2b66f\" (UID: \"7672fd63-163a-481c-aaf8-a1447da0fbf0\") " pod="openshift-image-registry/image-registry-5f57555bf7-2b66f"
Sep 18 19:38:56.990971 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.990941    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7672fd63-163a-481c-aaf8-a1447da0fbf0-image-registry-private-configuration\") pod \"image-registry-5f57555bf7-2b66f\" (UID: \"7672fd63-163a-481c-aaf8-a1447da0fbf0\") " pod="openshift-image-registry/image-registry-5f57555bf7-2b66f"
Sep 18 19:38:56.991181 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.991164    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/945a444e-9381-4e2f-b4c1-64be0ec88e6c-serving-cert\") pod \"console-operator-fd4bb46c-58vl4\" (UID: \"945a444e-9381-4e2f-b4c1-64be0ec88e6c\") " pod="openshift-console-operator/console-operator-fd4bb46c-58vl4"
Sep 18 19:38:56.991225 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.991210    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7672fd63-163a-481c-aaf8-a1447da0fbf0-installation-pull-secrets\") pod \"image-registry-5f57555bf7-2b66f\" (UID: \"7672fd63-163a-481c-aaf8-a1447da0fbf0\") " pod="openshift-image-registry/image-registry-5f57555bf7-2b66f"
Sep 18 19:38:56.996878 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.996851    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzzl2\" (UniqueName: \"kubernetes.io/projected/18f89927-0873-43f7-b237-ec56aaacb7c0-kube-api-access-nzzl2\") pod \"cluster-samples-operator-685d7d68b5-w4mzh\" (UID: \"18f89927-0873-43f7-b237-ec56aaacb7c0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-w4mzh"
Sep 18 19:38:56.997181 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.997163    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6nz8\" (UniqueName: \"kubernetes.io/projected/945a444e-9381-4e2f-b4c1-64be0ec88e6c-kube-api-access-w6nz8\") pod \"console-operator-fd4bb46c-58vl4\" (UID: \"945a444e-9381-4e2f-b4c1-64be0ec88e6c\") " pod="openshift-console-operator/console-operator-fd4bb46c-58vl4"
Sep 18 19:38:56.997336 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.997319    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qg4x\" (UniqueName: \"kubernetes.io/projected/7672fd63-163a-481c-aaf8-a1447da0fbf0-kube-api-access-5qg4x\") pod \"image-registry-5f57555bf7-2b66f\" (UID: \"7672fd63-163a-481c-aaf8-a1447da0fbf0\") " pod="openshift-image-registry/image-registry-5f57555bf7-2b66f"
Sep 18 19:38:56.997407 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:56.997389    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7672fd63-163a-481c-aaf8-a1447da0fbf0-bound-sa-token\") pod \"image-registry-5f57555bf7-2b66f\" (UID: \"7672fd63-163a-481c-aaf8-a1447da0fbf0\") " pod="openshift-image-registry/image-registry-5f57555bf7-2b66f"
Sep 18 19:38:57.053453 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:57.053376    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-58vl4"
Sep 18 19:38:57.111615 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:57.111582    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-lshmd"]
Sep 18 19:38:57.115212 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:57.115188    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b05425f_3a0e_40b6_b1af_4f3098050245.slice/crio-f04ac4fd657ce9b04bec680625a2630979e256bb6164e47cb80e5b48bcf7d2e5 WatchSource:0}: Error finding container f04ac4fd657ce9b04bec680625a2630979e256bb6164e47cb80e5b48bcf7d2e5: Status 404 returned error can't find the container with id f04ac4fd657ce9b04bec680625a2630979e256bb6164e47cb80e5b48bcf7d2e5
Sep 18 19:38:57.320849 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:57.320764    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-58vl4"]
Sep 18 19:38:57.323889 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:38:57.323859    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod945a444e_9381_4e2f_b4c1_64be0ec88e6c.slice/crio-0c80ec662ede3e5e4e44126ff7a4091901f5b3ba4f46736c7d1a89559c73afae WatchSource:0}: Error finding container 0c80ec662ede3e5e4e44126ff7a4091901f5b3ba4f46736c7d1a89559c73afae: Status 404 returned error can't find the container with id 0c80ec662ede3e5e4e44126ff7a4091901f5b3ba4f46736c7d1a89559c73afae
Sep 18 19:38:57.385443 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:57.385405    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-58vl4" event={"ID":"945a444e-9381-4e2f-b4c1-64be0ec88e6c","Type":"ContainerStarted","Data":"0c80ec662ede3e5e4e44126ff7a4091901f5b3ba4f46736c7d1a89559c73afae"}
Sep 18 19:38:57.386377 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:57.386352    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-lshmd" event={"ID":"9b05425f-3a0e-40b6-b1af-4f3098050245","Type":"ContainerStarted","Data":"f04ac4fd657ce9b04bec680625a2630979e256bb6164e47cb80e5b48bcf7d2e5"}
Sep 18 19:38:57.392726 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:57.392697    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20027a0c-0095-4418-aaa4-d0a6f273ef85-metrics-certs\") pod \"router-default-756fcb9d6-47xpw\" (UID: \"20027a0c-0095-4418-aaa4-d0a6f273ef85\") " pod="openshift-ingress/router-default-756fcb9d6-47xpw"
Sep 18 19:38:57.392846 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:57.392831    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20027a0c-0095-4418-aaa4-d0a6f273ef85-service-ca-bundle\") pod \"router-default-756fcb9d6-47xpw\" (UID: \"20027a0c-0095-4418-aaa4-d0a6f273ef85\") " pod="openshift-ingress/router-default-756fcb9d6-47xpw"
Sep 18 19:38:57.392846 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:57.392840    2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 19:38:57.392948 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:57.392894    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20027a0c-0095-4418-aaa4-d0a6f273ef85-metrics-certs podName:20027a0c-0095-4418-aaa4-d0a6f273ef85 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:58.392877515 +0000 UTC m=+54.903233584 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/20027a0c-0095-4418-aaa4-d0a6f273ef85-metrics-certs") pod "router-default-756fcb9d6-47xpw" (UID: "20027a0c-0095-4418-aaa4-d0a6f273ef85") : secret "router-metrics-certs-default" not found
Sep 18 19:38:57.393030 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:57.392972    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/20027a0c-0095-4418-aaa4-d0a6f273ef85-service-ca-bundle podName:20027a0c-0095-4418-aaa4-d0a6f273ef85 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:58.392954969 +0000 UTC m=+54.903311056 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/20027a0c-0095-4418-aaa4-d0a6f273ef85-service-ca-bundle") pod "router-default-756fcb9d6-47xpw" (UID: "20027a0c-0095-4418-aaa4-d0a6f273ef85") : configmap references non-existent config key: service-ca.crt
Sep 18 19:38:57.494177 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:57.494137    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7672fd63-163a-481c-aaf8-a1447da0fbf0-registry-tls\") pod \"image-registry-5f57555bf7-2b66f\" (UID: \"7672fd63-163a-481c-aaf8-a1447da0fbf0\") " pod="openshift-image-registry/image-registry-5f57555bf7-2b66f"
Sep 18 19:38:57.494360 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:57.494215    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/18f89927-0873-43f7-b237-ec56aaacb7c0-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-w4mzh\" (UID: \"18f89927-0873-43f7-b237-ec56aaacb7c0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-w4mzh"
Sep 18 19:38:57.494360 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:57.494285    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 19:38:57.494360 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:57.494306    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f57555bf7-2b66f: secret "image-registry-tls" not found
Sep 18 19:38:57.494360 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:57.494322    2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 19:38:57.494505 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:57.494368    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7672fd63-163a-481c-aaf8-a1447da0fbf0-registry-tls podName:7672fd63-163a-481c-aaf8-a1447da0fbf0 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:58.494353302 +0000 UTC m=+55.004709375 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7672fd63-163a-481c-aaf8-a1447da0fbf0-registry-tls") pod "image-registry-5f57555bf7-2b66f" (UID: "7672fd63-163a-481c-aaf8-a1447da0fbf0") : secret "image-registry-tls" not found
Sep 18 19:38:57.494505 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:57.494380    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18f89927-0873-43f7-b237-ec56aaacb7c0-samples-operator-tls podName:18f89927-0873-43f7-b237-ec56aaacb7c0 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:58.494374808 +0000 UTC m=+55.004730877 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/18f89927-0873-43f7-b237-ec56aaacb7c0-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-w4mzh" (UID: "18f89927-0873-43f7-b237-ec56aaacb7c0") : secret "samples-operator-tls" not found
Sep 18 19:38:57.845864 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:57.845782    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:57.899866 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:57.899822    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:58.401381 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:58.401342    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20027a0c-0095-4418-aaa4-d0a6f273ef85-service-ca-bundle\") pod \"router-default-756fcb9d6-47xpw\" (UID: \"20027a0c-0095-4418-aaa4-d0a6f273ef85\") " pod="openshift-ingress/router-default-756fcb9d6-47xpw"
Sep 18 19:38:58.401623 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:58.401508    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20027a0c-0095-4418-aaa4-d0a6f273ef85-metrics-certs\") pod \"router-default-756fcb9d6-47xpw\" (UID: \"20027a0c-0095-4418-aaa4-d0a6f273ef85\") " pod="openshift-ingress/router-default-756fcb9d6-47xpw"
Sep 18 19:38:58.401623 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:58.401528    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/20027a0c-0095-4418-aaa4-d0a6f273ef85-service-ca-bundle podName:20027a0c-0095-4418-aaa4-d0a6f273ef85 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:00.40150296 +0000 UTC m=+56.911859048 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/20027a0c-0095-4418-aaa4-d0a6f273ef85-service-ca-bundle") pod "router-default-756fcb9d6-47xpw" (UID: "20027a0c-0095-4418-aaa4-d0a6f273ef85") : configmap references non-existent config key: service-ca.crt
Sep 18 19:38:58.401623 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:58.401607    2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 19:38:58.401807 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:58.401660    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20027a0c-0095-4418-aaa4-d0a6f273ef85-metrics-certs podName:20027a0c-0095-4418-aaa4-d0a6f273ef85 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:00.401645309 +0000 UTC m=+56.912001385 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/20027a0c-0095-4418-aaa4-d0a6f273ef85-metrics-certs") pod "router-default-756fcb9d6-47xpw" (UID: "20027a0c-0095-4418-aaa4-d0a6f273ef85") : secret "router-metrics-certs-default" not found
Sep 18 19:38:58.502764 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:58.502724    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/18f89927-0873-43f7-b237-ec56aaacb7c0-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-w4mzh\" (UID: \"18f89927-0873-43f7-b237-ec56aaacb7c0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-w4mzh"
Sep 18 19:38:58.502969 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:58.502839    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7672fd63-163a-481c-aaf8-a1447da0fbf0-registry-tls\") pod \"image-registry-5f57555bf7-2b66f\" (UID: \"7672fd63-163a-481c-aaf8-a1447da0fbf0\") " pod="openshift-image-registry/image-registry-5f57555bf7-2b66f"
Sep 18 19:38:58.502969 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:58.502916    2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 19:38:58.503101 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:58.502965    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 19:38:58.503101 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:58.503001    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f57555bf7-2b66f: secret "image-registry-tls" not found
Sep 18 19:38:58.503101 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:58.503015    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18f89927-0873-43f7-b237-ec56aaacb7c0-samples-operator-tls podName:18f89927-0873-43f7-b237-ec56aaacb7c0 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:00.50297642 +0000 UTC m=+57.013332512 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/18f89927-0873-43f7-b237-ec56aaacb7c0-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-w4mzh" (UID: "18f89927-0873-43f7-b237-ec56aaacb7c0") : secret "samples-operator-tls" not found
Sep 18 19:38:58.503101 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:38:58.503057    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7672fd63-163a-481c-aaf8-a1447da0fbf0-registry-tls podName:7672fd63-163a-481c-aaf8-a1447da0fbf0 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:00.50303887 +0000 UTC m=+57.013394971 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7672fd63-163a-481c-aaf8-a1447da0fbf0-registry-tls") pod "image-registry-5f57555bf7-2b66f" (UID: "7672fd63-163a-481c-aaf8-a1447da0fbf0") : secret "image-registry-tls" not found
Sep 18 19:38:58.883133 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:58.883084    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:58.984036 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:58.983996    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:59.930965 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:38:59.930870    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:00.396324 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:00.396284    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-lshmd" event={"ID":"9b05425f-3a0e-40b6-b1af-4f3098050245","Type":"ContainerStarted","Data":"9651a802fec1232c271830bfdca7ebe0e25c80b29d1053b1f826db77406e3a24"}
Sep 18 19:39:00.398155 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:00.398127    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-58vl4_945a444e-9381-4e2f-b4c1-64be0ec88e6c/console-operator/0.log"
Sep 18 19:39:00.398290 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:00.398176    2564 generic.go:356] "Generic (PLEG): container finished" podID="945a444e-9381-4e2f-b4c1-64be0ec88e6c" containerID="c38c5fecd9e63ba3398c88e1d5bec66f927229b75a3c7f8097bc7fe5e357479b" exitCode=255
Sep 18 19:39:00.398290 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:00.398217    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-58vl4" event={"ID":"945a444e-9381-4e2f-b4c1-64be0ec88e6c","Type":"ContainerDied","Data":"c38c5fecd9e63ba3398c88e1d5bec66f927229b75a3c7f8097bc7fe5e357479b"}
Sep 18 19:39:00.398450 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:00.398433    2564 scope.go:117] "RemoveContainer" containerID="c38c5fecd9e63ba3398c88e1d5bec66f927229b75a3c7f8097bc7fe5e357479b"
Sep 18 19:39:00.411093 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:00.411040    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-lshmd" podStartSLOduration=1.883571308 podStartE2EDuration="4.411017911s" totalImagesPullingTime="2.527446603s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:38:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:00.410068735 +0000 UTC m=+56.920424848" watchObservedRunningTime="2026-09-18 19:39:00.411017911 +0000 UTC m=+56.921374005"
Sep 18 19:39:00.420250 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:00.420210    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20027a0c-0095-4418-aaa4-d0a6f273ef85-metrics-certs\") pod \"router-default-756fcb9d6-47xpw\" (UID: \"20027a0c-0095-4418-aaa4-d0a6f273ef85\") " pod="openshift-ingress/router-default-756fcb9d6-47xpw"
Sep 18 19:39:00.420442 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:39:00.420414    2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 19:39:00.420883 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:39:00.420492    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20027a0c-0095-4418-aaa4-d0a6f273ef85-metrics-certs podName:20027a0c-0095-4418-aaa4-d0a6f273ef85 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:04.420469766 +0000 UTC m=+60.930825852 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/20027a0c-0095-4418-aaa4-d0a6f273ef85-metrics-certs") pod "router-default-756fcb9d6-47xpw" (UID: "20027a0c-0095-4418-aaa4-d0a6f273ef85") : secret "router-metrics-certs-default" not found
Sep 18 19:39:00.420996 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:00.420905    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20027a0c-0095-4418-aaa4-d0a6f273ef85-service-ca-bundle\") pod \"router-default-756fcb9d6-47xpw\" (UID: \"20027a0c-0095-4418-aaa4-d0a6f273ef85\") " pod="openshift-ingress/router-default-756fcb9d6-47xpw"
Sep 18 19:39:00.421107 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:39:00.421091    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/20027a0c-0095-4418-aaa4-d0a6f273ef85-service-ca-bundle podName:20027a0c-0095-4418-aaa4-d0a6f273ef85 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:04.421078151 +0000 UTC m=+60.931434221 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/20027a0c-0095-4418-aaa4-d0a6f273ef85-service-ca-bundle") pod "router-default-756fcb9d6-47xpw" (UID: "20027a0c-0095-4418-aaa4-d0a6f273ef85") : configmap references non-existent config key: service-ca.crt
Sep 18 19:39:00.522065 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:00.522038    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7672fd63-163a-481c-aaf8-a1447da0fbf0-registry-tls\") pod \"image-registry-5f57555bf7-2b66f\" (UID: \"7672fd63-163a-481c-aaf8-a1447da0fbf0\") " pod="openshift-image-registry/image-registry-5f57555bf7-2b66f"
Sep 18 19:39:00.522202 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:39:00.522183    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 19:39:00.522239 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:39:00.522207    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f57555bf7-2b66f: secret "image-registry-tls" not found
Sep 18 19:39:00.522276 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:00.522186    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/18f89927-0873-43f7-b237-ec56aaacb7c0-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-w4mzh\" (UID: \"18f89927-0873-43f7-b237-ec56aaacb7c0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-w4mzh"
Sep 18 19:39:00.522276 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:39:00.522257    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7672fd63-163a-481c-aaf8-a1447da0fbf0-registry-tls podName:7672fd63-163a-481c-aaf8-a1447da0fbf0 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:04.522243224 +0000 UTC m=+61.032599293 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7672fd63-163a-481c-aaf8-a1447da0fbf0-registry-tls") pod "image-registry-5f57555bf7-2b66f" (UID: "7672fd63-163a-481c-aaf8-a1447da0fbf0") : secret "image-registry-tls" not found
Sep 18 19:39:00.522276 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:39:00.522257    2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 19:39:00.522363 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:39:00.522312    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18f89927-0873-43f7-b237-ec56aaacb7c0-samples-operator-tls podName:18f89927-0873-43f7-b237-ec56aaacb7c0 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:04.522304121 +0000 UTC m=+61.032660206 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/18f89927-0873-43f7-b237-ec56aaacb7c0-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-w4mzh" (UID: "18f89927-0873-43f7-b237-ec56aaacb7c0") : secret "samples-operator-tls" not found
Sep 18 19:39:01.402688 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:01.402660    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-58vl4_945a444e-9381-4e2f-b4c1-64be0ec88e6c/console-operator/1.log"
Sep 18 19:39:01.403142 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:01.403126    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-58vl4_945a444e-9381-4e2f-b4c1-64be0ec88e6c/console-operator/0.log"
Sep 18 19:39:01.403186 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:01.403165    2564 generic.go:356] "Generic (PLEG): container finished" podID="945a444e-9381-4e2f-b4c1-64be0ec88e6c" containerID="8600f4eb75644be5811b55bf1c8d09e85718e26301bd7d3ee3ce0e5fedfbf263" exitCode=255
Sep 18 19:39:01.403220 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:01.403196    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-58vl4" event={"ID":"945a444e-9381-4e2f-b4c1-64be0ec88e6c","Type":"ContainerDied","Data":"8600f4eb75644be5811b55bf1c8d09e85718e26301bd7d3ee3ce0e5fedfbf263"}
Sep 18 19:39:01.403269 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:01.403257    2564 scope.go:117] "RemoveContainer" containerID="c38c5fecd9e63ba3398c88e1d5bec66f927229b75a3c7f8097bc7fe5e357479b"
Sep 18 19:39:01.403540 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:01.403519    2564 scope.go:117] "RemoveContainer" containerID="8600f4eb75644be5811b55bf1c8d09e85718e26301bd7d3ee3ce0e5fedfbf263"
Sep 18 19:39:01.403729 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:39:01.403708    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-58vl4_openshift-console-operator(945a444e-9381-4e2f-b4c1-64be0ec88e6c)\"" pod="openshift-console-operator/console-operator-fd4bb46c-58vl4" podUID="945a444e-9381-4e2f-b4c1-64be0ec88e6c"
Sep 18 19:39:01.455670 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:01.455637    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:01.556454 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:01.556411    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:02.051226 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:02.051195    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-jfgj8"]
Sep 18 19:39:02.054097 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:02.054076    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jfgj8"
Sep 18 19:39:02.056274 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:02.056247    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 18 19:39:02.056700 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:02.056680    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 18 19:39:02.056775 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:02.056710    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 18 19:39:02.057114 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:02.057098    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 18 19:39:02.057180 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:02.057148    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-696cc\""
Sep 18 19:39:02.066285 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:02.066263    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jfgj8"]
Sep 18 19:39:02.074654 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:02.074629    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:02.135960 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:02.135926    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2c9bbaf8-3c60-402a-a13f-03d8e012c7c1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jfgj8\" (UID: \"2c9bbaf8-3c60-402a-a13f-03d8e012c7c1\") " pod="openshift-insights/insights-runtime-extractor-jfgj8"
Sep 18 19:39:02.136166 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:02.135971    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2c9bbaf8-3c60-402a-a13f-03d8e012c7c1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jfgj8\" (UID: \"2c9bbaf8-3c60-402a-a13f-03d8e012c7c1\") " pod="openshift-insights/insights-runtime-extractor-jfgj8"
Sep 18 19:39:02.136166 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:02.136108    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2c9bbaf8-3c60-402a-a13f-03d8e012c7c1-data-volume\") pod \"insights-runtime-extractor-jfgj8\" (UID: \"2c9bbaf8-3c60-402a-a13f-03d8e012c7c1\") " pod="openshift-insights/insights-runtime-extractor-jfgj8"
Sep 18 19:39:02.136288 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:02.136166    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-744j7\" (UniqueName: \"kubernetes.io/projected/2c9bbaf8-3c60-402a-a13f-03d8e012c7c1-kube-api-access-744j7\") pod \"insights-runtime-extractor-jfgj8\" (UID: \"2c9bbaf8-3c60-402a-a13f-03d8e012c7c1\") " pod="openshift-insights/insights-runtime-extractor-jfgj8"
Sep 18 19:39:02.136288 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:02.136273    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2c9bbaf8-3c60-402a-a13f-03d8e012c7c1-crio-socket\") pod \"insights-runtime-extractor-jfgj8\" (UID: \"2c9bbaf8-3c60-402a-a13f-03d8e012c7c1\") " pod="openshift-insights/insights-runtime-extractor-jfgj8"
Sep 18 19:39:02.145671 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:02.145638    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:02.237107 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:02.237057    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2c9bbaf8-3c60-402a-a13f-03d8e012c7c1-data-volume\") pod \"insights-runtime-extractor-jfgj8\" (UID: \"2c9bbaf8-3c60-402a-a13f-03d8e012c7c1\") " pod="openshift-insights/insights-runtime-extractor-jfgj8"
Sep 18 19:39:02.237107 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:02.237111    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-744j7\" (UniqueName: \"kubernetes.io/projected/2c9bbaf8-3c60-402a-a13f-03d8e012c7c1-kube-api-access-744j7\") pod \"insights-runtime-extractor-jfgj8\" (UID: \"2c9bbaf8-3c60-402a-a13f-03d8e012c7c1\") " pod="openshift-insights/insights-runtime-extractor-jfgj8"
Sep 18 19:39:02.237358 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:02.237171    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2c9bbaf8-3c60-402a-a13f-03d8e012c7c1-crio-socket\") pod \"insights-runtime-extractor-jfgj8\" (UID: \"2c9bbaf8-3c60-402a-a13f-03d8e012c7c1\") " pod="openshift-insights/insights-runtime-extractor-jfgj8"
Sep 18 19:39:02.237358 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:02.237208    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2c9bbaf8-3c60-402a-a13f-03d8e012c7c1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jfgj8\" (UID: \"2c9bbaf8-3c60-402a-a13f-03d8e012c7c1\") " pod="openshift-insights/insights-runtime-extractor-jfgj8"
Sep 18 19:39:02.237358 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:02.237233    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2c9bbaf8-3c60-402a-a13f-03d8e012c7c1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jfgj8\" (UID: \"2c9bbaf8-3c60-402a-a13f-03d8e012c7c1\") " pod="openshift-insights/insights-runtime-extractor-jfgj8"
Sep 18 19:39:02.237358 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:02.237290    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2c9bbaf8-3c60-402a-a13f-03d8e012c7c1-crio-socket\") pod \"insights-runtime-extractor-jfgj8\" (UID: \"2c9bbaf8-3c60-402a-a13f-03d8e012c7c1\") " pod="openshift-insights/insights-runtime-extractor-jfgj8"
Sep 18 19:39:02.237358 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:39:02.237332    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:39:02.237358 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:39:02.237343    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:39:02.237562 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:39:02.237382    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c9bbaf8-3c60-402a-a13f-03d8e012c7c1-insights-runtime-extractor-tls podName:2c9bbaf8-3c60-402a-a13f-03d8e012c7c1 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:02.737369306 +0000 UTC m=+59.247725374 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2c9bbaf8-3c60-402a-a13f-03d8e012c7c1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jfgj8" (UID: "2c9bbaf8-3c60-402a-a13f-03d8e012c7c1") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:39:02.237562 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:39:02.237393    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2c9bbaf8-3c60-402a-a13f-03d8e012c7c1-kube-rbac-proxy-cm podName:2c9bbaf8-3c60-402a-a13f-03d8e012c7c1 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:02.737387858 +0000 UTC m=+59.247743927 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2c9bbaf8-3c60-402a-a13f-03d8e012c7c1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jfgj8" (UID: "2c9bbaf8-3c60-402a-a13f-03d8e012c7c1") : configmap "kube-rbac-proxy" not found
Sep 18 19:39:02.237562 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:02.237456    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2c9bbaf8-3c60-402a-a13f-03d8e012c7c1-data-volume\") pod \"insights-runtime-extractor-jfgj8\" (UID: \"2c9bbaf8-3c60-402a-a13f-03d8e012c7c1\") " pod="openshift-insights/insights-runtime-extractor-jfgj8"
Sep 18 19:39:02.240748 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:02.240720    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:02.245056 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:02.245036    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-744j7\" (UniqueName: \"kubernetes.io/projected/2c9bbaf8-3c60-402a-a13f-03d8e012c7c1-kube-api-access-744j7\") pod \"insights-runtime-extractor-jfgj8\" (UID: \"2c9bbaf8-3c60-402a-a13f-03d8e012c7c1\") " pod="openshift-insights/insights-runtime-extractor-jfgj8"
Sep 18 19:39:02.407849 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:02.407824    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-58vl4_945a444e-9381-4e2f-b4c1-64be0ec88e6c/console-operator/1.log"
Sep 18 19:39:02.408237 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:02.408199    2564 scope.go:117] "RemoveContainer" containerID="8600f4eb75644be5811b55bf1c8d09e85718e26301bd7d3ee3ce0e5fedfbf263"
Sep 18 19:39:02.408378 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:39:02.408356    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-58vl4_openshift-console-operator(945a444e-9381-4e2f-b4c1-64be0ec88e6c)\"" pod="openshift-console-operator/console-operator-fd4bb46c-58vl4" podUID="945a444e-9381-4e2f-b4c1-64be0ec88e6c"
Sep 18 19:39:02.742239 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:02.742139    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2c9bbaf8-3c60-402a-a13f-03d8e012c7c1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jfgj8\" (UID: \"2c9bbaf8-3c60-402a-a13f-03d8e012c7c1\") " pod="openshift-insights/insights-runtime-extractor-jfgj8"
Sep 18 19:39:02.742239 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:02.742198    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2c9bbaf8-3c60-402a-a13f-03d8e012c7c1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jfgj8\" (UID: \"2c9bbaf8-3c60-402a-a13f-03d8e012c7c1\") " pod="openshift-insights/insights-runtime-extractor-jfgj8"
Sep 18 19:39:02.742480 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:39:02.742271    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:39:02.742480 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:39:02.742348    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2c9bbaf8-3c60-402a-a13f-03d8e012c7c1-kube-rbac-proxy-cm podName:2c9bbaf8-3c60-402a-a13f-03d8e012c7c1 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:03.742329982 +0000 UTC m=+60.252686051 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2c9bbaf8-3c60-402a-a13f-03d8e012c7c1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jfgj8" (UID: "2c9bbaf8-3c60-402a-a13f-03d8e012c7c1") : configmap "kube-rbac-proxy" not found
Sep 18 19:39:02.742480 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:39:02.742400    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:39:02.742480 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:39:02.742460    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c9bbaf8-3c60-402a-a13f-03d8e012c7c1-insights-runtime-extractor-tls podName:2c9bbaf8-3c60-402a-a13f-03d8e012c7c1 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:03.742439169 +0000 UTC m=+60.252795237 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2c9bbaf8-3c60-402a-a13f-03d8e012c7c1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jfgj8" (UID: "2c9bbaf8-3c60-402a-a13f-03d8e012c7c1") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:39:02.887871 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:02.887828    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:02.968567 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:02.968524    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:03.021598 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:03.021511    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:03.325002 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:03.324918    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xcbfz"
Sep 18 19:39:03.471085 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:03.471054    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:03.571725 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:03.571688    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:03.751623 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:03.751585    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2c9bbaf8-3c60-402a-a13f-03d8e012c7c1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jfgj8\" (UID: \"2c9bbaf8-3c60-402a-a13f-03d8e012c7c1\") " pod="openshift-insights/insights-runtime-extractor-jfgj8"
Sep 18 19:39:03.751781 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:03.751634    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2c9bbaf8-3c60-402a-a13f-03d8e012c7c1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jfgj8\" (UID: \"2c9bbaf8-3c60-402a-a13f-03d8e012c7c1\") " pod="openshift-insights/insights-runtime-extractor-jfgj8"
Sep 18 19:39:03.751781 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:39:03.751717    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:39:03.751781 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:39:03.751739    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:39:03.751781 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:39:03.751782    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2c9bbaf8-3c60-402a-a13f-03d8e012c7c1-kube-rbac-proxy-cm podName:2c9bbaf8-3c60-402a-a13f-03d8e012c7c1 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:05.75176866 +0000 UTC m=+62.262124730 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2c9bbaf8-3c60-402a-a13f-03d8e012c7c1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jfgj8" (UID: "2c9bbaf8-3c60-402a-a13f-03d8e012c7c1") : configmap "kube-rbac-proxy" not found
Sep 18 19:39:03.751958 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:39:03.751849    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c9bbaf8-3c60-402a-a13f-03d8e012c7c1-insights-runtime-extractor-tls podName:2c9bbaf8-3c60-402a-a13f-03d8e012c7c1 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:05.75183236 +0000 UTC m=+62.262188433 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2c9bbaf8-3c60-402a-a13f-03d8e012c7c1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jfgj8" (UID: "2c9bbaf8-3c60-402a-a13f-03d8e012c7c1") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:39:03.796817 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:03.796786    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:03.822337 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:03.822306    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dwwf5_3f7e2771-a1bc-4ac9-9b67-1c7c9473d895/dns-node-resolver/0.log"
Sep 18 19:39:03.898643 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:03.898596    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:04.459320 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:04.459278    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20027a0c-0095-4418-aaa4-d0a6f273ef85-metrics-certs\") pod \"router-default-756fcb9d6-47xpw\" (UID: \"20027a0c-0095-4418-aaa4-d0a6f273ef85\") " pod="openshift-ingress/router-default-756fcb9d6-47xpw"
Sep 18 19:39:04.459320 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:04.459323    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20027a0c-0095-4418-aaa4-d0a6f273ef85-service-ca-bundle\") pod \"router-default-756fcb9d6-47xpw\" (UID: \"20027a0c-0095-4418-aaa4-d0a6f273ef85\") " pod="openshift-ingress/router-default-756fcb9d6-47xpw"
Sep 18 19:39:04.459626 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:39:04.459451    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/20027a0c-0095-4418-aaa4-d0a6f273ef85-service-ca-bundle podName:20027a0c-0095-4418-aaa4-d0a6f273ef85 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:12.459435485 +0000 UTC m=+68.969791555 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/20027a0c-0095-4418-aaa4-d0a6f273ef85-service-ca-bundle") pod "router-default-756fcb9d6-47xpw" (UID: "20027a0c-0095-4418-aaa4-d0a6f273ef85") : configmap references non-existent config key: service-ca.crt
Sep 18 19:39:04.459626 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:39:04.459506    2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 19:39:04.459626 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:39:04.459554    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20027a0c-0095-4418-aaa4-d0a6f273ef85-metrics-certs podName:20027a0c-0095-4418-aaa4-d0a6f273ef85 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:12.459540721 +0000 UTC m=+68.969896802 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/20027a0c-0095-4418-aaa4-d0a6f273ef85-metrics-certs") pod "router-default-756fcb9d6-47xpw" (UID: "20027a0c-0095-4418-aaa4-d0a6f273ef85") : secret "router-metrics-certs-default" not found
Sep 18 19:39:04.560195 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:04.560159    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7672fd63-163a-481c-aaf8-a1447da0fbf0-registry-tls\") pod \"image-registry-5f57555bf7-2b66f\" (UID: \"7672fd63-163a-481c-aaf8-a1447da0fbf0\") " pod="openshift-image-registry/image-registry-5f57555bf7-2b66f"
Sep 18 19:39:04.560601 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:04.560219    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/18f89927-0873-43f7-b237-ec56aaacb7c0-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-w4mzh\" (UID: \"18f89927-0873-43f7-b237-ec56aaacb7c0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-w4mzh"
Sep 18 19:39:04.560601 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:39:04.560314    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 19:39:04.560601 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:39:04.560340    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f57555bf7-2b66f: secret "image-registry-tls" not found
Sep 18 19:39:04.560601 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:39:04.560322    2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 19:39:04.560601 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:39:04.560393    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7672fd63-163a-481c-aaf8-a1447da0fbf0-registry-tls podName:7672fd63-163a-481c-aaf8-a1447da0fbf0 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:12.56037853 +0000 UTC m=+69.070734599 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7672fd63-163a-481c-aaf8-a1447da0fbf0-registry-tls") pod "image-registry-5f57555bf7-2b66f" (UID: "7672fd63-163a-481c-aaf8-a1447da0fbf0") : secret "image-registry-tls" not found
Sep 18 19:39:04.560601 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:39:04.560406    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18f89927-0873-43f7-b237-ec56aaacb7c0-samples-operator-tls podName:18f89927-0873-43f7-b237-ec56aaacb7c0 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:12.560399975 +0000 UTC m=+69.070756043 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/18f89927-0873-43f7-b237-ec56aaacb7c0-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-w4mzh" (UID: "18f89927-0873-43f7-b237-ec56aaacb7c0") : secret "samples-operator-tls" not found
Sep 18 19:39:04.967725 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:04.967692    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:05.022392 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:05.022366    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8hdsx_71862b1f-4124-47a1-8942-60acbbb5bdfb/node-ca/0.log"
Sep 18 19:39:05.383897 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:05.383864    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:05.623151 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:05.623122    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-d8v8m_218f155a-deaf-4ea6-8234-74b776e05a7b/serve-healthcheck-canary/0.log"
Sep 18 19:39:05.757217 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:05.757140    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:05.771681 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:05.771644    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2c9bbaf8-3c60-402a-a13f-03d8e012c7c1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jfgj8\" (UID: \"2c9bbaf8-3c60-402a-a13f-03d8e012c7c1\") " pod="openshift-insights/insights-runtime-extractor-jfgj8"
Sep 18 19:39:05.771822 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:05.771694    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2c9bbaf8-3c60-402a-a13f-03d8e012c7c1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jfgj8\" (UID: \"2c9bbaf8-3c60-402a-a13f-03d8e012c7c1\") " pod="openshift-insights/insights-runtime-extractor-jfgj8"
Sep 18 19:39:05.771822 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:39:05.771766    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:39:05.771900 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:39:05.771824    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:39:05.771900 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:39:05.771831    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2c9bbaf8-3c60-402a-a13f-03d8e012c7c1-kube-rbac-proxy-cm podName:2c9bbaf8-3c60-402a-a13f-03d8e012c7c1 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:09.771815217 +0000 UTC m=+66.282171290 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2c9bbaf8-3c60-402a-a13f-03d8e012c7c1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jfgj8" (UID: "2c9bbaf8-3c60-402a-a13f-03d8e012c7c1") : configmap "kube-rbac-proxy" not found
Sep 18 19:39:05.771900 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:39:05.771865    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c9bbaf8-3c60-402a-a13f-03d8e012c7c1-insights-runtime-extractor-tls podName:2c9bbaf8-3c60-402a-a13f-03d8e012c7c1 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:09.771854134 +0000 UTC m=+66.282210220 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2c9bbaf8-3c60-402a-a13f-03d8e012c7c1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jfgj8" (UID: "2c9bbaf8-3c60-402a-a13f-03d8e012c7c1") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:39:06.156444 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:06.156404    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:06.213751 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:06.213708    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:07.053538 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:07.053500    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-58vl4"
Sep 18 19:39:07.053945 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:07.053559    2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-58vl4"
Sep 18 19:39:07.053945 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:07.053893    2564 scope.go:117] "RemoveContainer" containerID="8600f4eb75644be5811b55bf1c8d09e85718e26301bd7d3ee3ce0e5fedfbf263"
Sep 18 19:39:07.054086 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:39:07.054065    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-58vl4_openshift-console-operator(945a444e-9381-4e2f-b4c1-64be0ec88e6c)\"" pod="openshift-console-operator/console-operator-fd4bb46c-58vl4" podUID="945a444e-9381-4e2f-b4c1-64be0ec88e6c"
Sep 18 19:39:08.096372 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:08.096340    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:08.215798 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:08.215761    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:08.316068 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:08.316036    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:08.420377 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:08.420347    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:08.477597 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:08.477559    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:08.608629 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:08.608595    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:08.676797 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:08.676711    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:08.701353 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:08.701308    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/256a72d8-e079-42fb-9426-c53adc283d1f-metrics-certs\") pod \"network-metrics-daemon-xgxbn\" (UID: \"256a72d8-e079-42fb-9426-c53adc283d1f\") " pod="openshift-multus/network-metrics-daemon-xgxbn"
Sep 18 19:39:08.703528 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:08.703509    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 19:39:08.712260 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:39:08.712237    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 18 19:39:08.712376 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:39:08.712300    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/256a72d8-e079-42fb-9426-c53adc283d1f-metrics-certs podName:256a72d8-e079-42fb-9426-c53adc283d1f nodeName:}" failed. No retries permitted until 2026-09-18 19:40:12.712281405 +0000 UTC m=+129.222637479 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/256a72d8-e079-42fb-9426-c53adc283d1f-metrics-certs") pod "network-metrics-daemon-xgxbn" (UID: "256a72d8-e079-42fb-9426-c53adc283d1f") : secret "metrics-daemon-secret" not found
Sep 18 19:39:08.777665 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:08.777629    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:08.801998 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:08.801943    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2939ca32-81a7-4e9d-bdc6-ffca73820918-original-pull-secret\") pod \"global-pull-secret-syncer-6vdpn\" (UID: \"2939ca32-81a7-4e9d-bdc6-ffca73820918\") " pod="kube-system/global-pull-secret-syncer-6vdpn"
Sep 18 19:39:08.804221 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:08.804202    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 19:39:08.815316 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:08.815291    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2939ca32-81a7-4e9d-bdc6-ffca73820918-original-pull-secret\") pod \"global-pull-secret-syncer-6vdpn\" (UID: \"2939ca32-81a7-4e9d-bdc6-ffca73820918\") " pod="kube-system/global-pull-secret-syncer-6vdpn"
Sep 18 19:39:08.902961 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:08.902924    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjgc5\" (UniqueName: \"kubernetes.io/projected/1b42bdad-4122-4a0a-978c-d2d5a7bdbd55-kube-api-access-zjgc5\") pod \"network-check-target-857bd\" (UID: \"1b42bdad-4122-4a0a-978c-d2d5a7bdbd55\") " pod="openshift-network-diagnostics/network-check-target-857bd"
Sep 18 19:39:08.905484 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:08.905460    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 19:39:08.915691 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:08.915672    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 19:39:08.919192 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:08.919173    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vdpn"
Sep 18 19:39:08.927246 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:08.927185    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjgc5\" (UniqueName: \"kubernetes.io/projected/1b42bdad-4122-4a0a-978c-d2d5a7bdbd55-kube-api-access-zjgc5\") pod \"network-check-target-857bd\" (UID: \"1b42bdad-4122-4a0a-978c-d2d5a7bdbd55\") " pod="openshift-network-diagnostics/network-check-target-857bd"
Sep 18 19:39:09.185296 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:09.185219    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6vdpn"]
Sep 18 19:39:09.188610 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:39:09.188582    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2939ca32_81a7_4e9d_bdc6_ffca73820918.slice/crio-b3d253afea625633e2aff69fe265cbd2fdbee54e1c368c33ac01b7b3f6911133 WatchSource:0}: Error finding container b3d253afea625633e2aff69fe265cbd2fdbee54e1c368c33ac01b7b3f6911133: Status 404 returned error can't find the container with id b3d253afea625633e2aff69fe265cbd2fdbee54e1c368c33ac01b7b3f6911133
Sep 18 19:39:09.208683 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:09.208655    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-pwxm5\""
Sep 18 19:39:09.217408 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:09.217378    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-857bd"
Sep 18 19:39:09.427154 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:09.427119    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6vdpn" event={"ID":"2939ca32-81a7-4e9d-bdc6-ffca73820918","Type":"ContainerStarted","Data":"b3d253afea625633e2aff69fe265cbd2fdbee54e1c368c33ac01b7b3f6911133"}
Sep 18 19:39:09.482850 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:09.482776    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-857bd"]
Sep 18 19:39:09.486017 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:39:09.485966    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b42bdad_4122_4a0a_978c_d2d5a7bdbd55.slice/crio-5099fe178c18045cccbbb48729764b8fbb02fa56446a83be5874e5a3f9828ce6 WatchSource:0}: Error finding container 5099fe178c18045cccbbb48729764b8fbb02fa56446a83be5874e5a3f9828ce6: Status 404 returned error can't find the container with id 5099fe178c18045cccbbb48729764b8fbb02fa56446a83be5874e5a3f9828ce6
Sep 18 19:39:09.771198 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:09.771097    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:09.811195 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:09.811163    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8b6ee93c-e19d-44f2-9ba9-ae133edb7686-metrics-tls\") pod \"dns-default-5krlr\" (UID: \"8b6ee93c-e19d-44f2-9ba9-ae133edb7686\") " pod="openshift-dns/dns-default-5krlr"
Sep 18 19:39:09.811400 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:09.811228    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2c9bbaf8-3c60-402a-a13f-03d8e012c7c1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jfgj8\" (UID: \"2c9bbaf8-3c60-402a-a13f-03d8e012c7c1\") " pod="openshift-insights/insights-runtime-extractor-jfgj8"
Sep 18 19:39:09.811400 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:09.811264    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2c9bbaf8-3c60-402a-a13f-03d8e012c7c1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jfgj8\" (UID: \"2c9bbaf8-3c60-402a-a13f-03d8e012c7c1\") " pod="openshift-insights/insights-runtime-extractor-jfgj8"
Sep 18 19:39:09.811400 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:39:09.811381    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:39:09.811400 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:39:09.811387    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:39:09.811584 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:39:09.811432    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:39:09.811584 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:39:09.811452    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2c9bbaf8-3c60-402a-a13f-03d8e012c7c1-kube-rbac-proxy-cm podName:2c9bbaf8-3c60-402a-a13f-03d8e012c7c1 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:17.811436432 +0000 UTC m=+74.321792513 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2c9bbaf8-3c60-402a-a13f-03d8e012c7c1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jfgj8" (UID: "2c9bbaf8-3c60-402a-a13f-03d8e012c7c1") : configmap "kube-rbac-proxy" not found
Sep 18 19:39:09.811584 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:39:09.811467    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b6ee93c-e19d-44f2-9ba9-ae133edb7686-metrics-tls podName:8b6ee93c-e19d-44f2-9ba9-ae133edb7686 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:41.811461274 +0000 UTC m=+98.321817343 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8b6ee93c-e19d-44f2-9ba9-ae133edb7686-metrics-tls") pod "dns-default-5krlr" (UID: "8b6ee93c-e19d-44f2-9ba9-ae133edb7686") : secret "dns-default-metrics-tls" not found
Sep 18 19:39:09.811584 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:39:09.811491    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c9bbaf8-3c60-402a-a13f-03d8e012c7c1-insights-runtime-extractor-tls podName:2c9bbaf8-3c60-402a-a13f-03d8e012c7c1 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:17.811472265 +0000 UTC m=+74.321828341 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2c9bbaf8-3c60-402a-a13f-03d8e012c7c1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jfgj8" (UID: "2c9bbaf8-3c60-402a-a13f-03d8e012c7c1") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:39:10.181468 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:10.181368    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:10.379241 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:10.379174    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:10.432356 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:10.432258    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-857bd" event={"ID":"1b42bdad-4122-4a0a-978c-d2d5a7bdbd55","Type":"ContainerStarted","Data":"5099fe178c18045cccbbb48729764b8fbb02fa56446a83be5874e5a3f9828ce6"}
Sep 18 19:39:10.583733 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:10.583645    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:10.779942 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:10.779853    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:11.771549 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:11.771503    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:11.826232 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:11.826192    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:12.178295 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:12.178256    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:12.380202 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:12.380166    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:12.537270 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:12.537188    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20027a0c-0095-4418-aaa4-d0a6f273ef85-metrics-certs\") pod \"router-default-756fcb9d6-47xpw\" (UID: \"20027a0c-0095-4418-aaa4-d0a6f273ef85\") " pod="openshift-ingress/router-default-756fcb9d6-47xpw"
Sep 18 19:39:12.537270 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:12.537227    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20027a0c-0095-4418-aaa4-d0a6f273ef85-service-ca-bundle\") pod \"router-default-756fcb9d6-47xpw\" (UID: \"20027a0c-0095-4418-aaa4-d0a6f273ef85\") " pod="openshift-ingress/router-default-756fcb9d6-47xpw"
Sep 18 19:39:12.537499 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:39:12.537386    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/20027a0c-0095-4418-aaa4-d0a6f273ef85-service-ca-bundle podName:20027a0c-0095-4418-aaa4-d0a6f273ef85 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:28.537371213 +0000 UTC m=+85.047727282 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/20027a0c-0095-4418-aaa4-d0a6f273ef85-service-ca-bundle") pod "router-default-756fcb9d6-47xpw" (UID: "20027a0c-0095-4418-aaa4-d0a6f273ef85") : configmap references non-existent config key: service-ca.crt
Sep 18 19:39:12.537499 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:39:12.537382    2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 19:39:12.537499 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:39:12.537467    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20027a0c-0095-4418-aaa4-d0a6f273ef85-metrics-certs podName:20027a0c-0095-4418-aaa4-d0a6f273ef85 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:28.537446911 +0000 UTC m=+85.047802997 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/20027a0c-0095-4418-aaa4-d0a6f273ef85-metrics-certs") pod "router-default-756fcb9d6-47xpw" (UID: "20027a0c-0095-4418-aaa4-d0a6f273ef85") : secret "router-metrics-certs-default" not found
Sep 18 19:39:12.581534 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:12.581487    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:12.637301 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:12.637266    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:12.637648 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:12.637618    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/18f89927-0873-43f7-b237-ec56aaacb7c0-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-w4mzh\" (UID: \"18f89927-0873-43f7-b237-ec56aaacb7c0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-w4mzh"
Sep 18 19:39:12.637794 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:12.637776    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7672fd63-163a-481c-aaf8-a1447da0fbf0-registry-tls\") pod \"image-registry-5f57555bf7-2b66f\" (UID: \"7672fd63-163a-481c-aaf8-a1447da0fbf0\") " pod="openshift-image-registry/image-registry-5f57555bf7-2b66f"
Sep 18 19:39:12.640451 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:12.640429    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7672fd63-163a-481c-aaf8-a1447da0fbf0-registry-tls\") pod \"image-registry-5f57555bf7-2b66f\" (UID: \"7672fd63-163a-481c-aaf8-a1447da0fbf0\") " pod="openshift-image-registry/image-registry-5f57555bf7-2b66f"
Sep 18 19:39:12.640570 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:12.640551    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/18f89927-0873-43f7-b237-ec56aaacb7c0-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-w4mzh\" (UID: \"18f89927-0873-43f7-b237-ec56aaacb7c0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-w4mzh"
Sep 18 19:39:12.662579 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:12.662547    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-x4hwn\""
Sep 18 19:39:12.667579 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:12.667553    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-ntrxj\""
Sep 18 19:39:12.671219 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:12.671188    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-w4mzh"
Sep 18 19:39:12.676434 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:12.676406    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f57555bf7-2b66f"
Sep 18 19:39:12.975302 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:12.975269    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:13.583686 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:13.583590    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:13.977057 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:13.976735    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:14.175779 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:14.175747    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:14.253087 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:14.252991    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-w4mzh"]
Sep 18 19:39:14.257556 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:14.257514    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5f57555bf7-2b66f"]
Sep 18 19:39:14.260968 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:39:14.260938    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7672fd63_163a_481c_aaf8_a1447da0fbf0.slice/crio-83b1412ebb63e831976b61fadf3391f1ff86460af0eb275ac54e0314062f0ec3 WatchSource:0}: Error finding container 83b1412ebb63e831976b61fadf3391f1ff86460af0eb275ac54e0314062f0ec3: Status 404 returned error can't find the container with id 83b1412ebb63e831976b61fadf3391f1ff86460af0eb275ac54e0314062f0ec3
Sep 18 19:39:14.362547 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:14.362508    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:14.445723 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:14.445690    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6vdpn" event={"ID":"2939ca32-81a7-4e9d-bdc6-ffca73820918","Type":"ContainerStarted","Data":"c7f0c6af4e1364ce584b95f9de791e0ee238a11c8312c2050a826955155eeabf"}
Sep 18 19:39:14.447394 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:14.447365    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-857bd" event={"ID":"1b42bdad-4122-4a0a-978c-d2d5a7bdbd55","Type":"ContainerStarted","Data":"545cb2fd1487311bae470ac312abfe95731d0af5d15722eedd32ccdf23d96bb4"}
Sep 18 19:39:14.447501 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:14.447432    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-857bd"
Sep 18 19:39:14.449344 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:14.449311    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f57555bf7-2b66f" event={"ID":"7672fd63-163a-481c-aaf8-a1447da0fbf0","Type":"ContainerStarted","Data":"230edfb702f4cc6a64daf5260df832a84165ee1a0bf66a608ce743cc760c4998"}
Sep 18 19:39:14.449459 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:14.449353    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f57555bf7-2b66f" event={"ID":"7672fd63-163a-481c-aaf8-a1447da0fbf0","Type":"ContainerStarted","Data":"83b1412ebb63e831976b61fadf3391f1ff86460af0eb275ac54e0314062f0ec3"}
Sep 18 19:39:14.449459 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:14.449376    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5f57555bf7-2b66f"
Sep 18 19:39:14.450512 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:14.450492    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-w4mzh" event={"ID":"18f89927-0873-43f7-b237-ec56aaacb7c0","Type":"ContainerStarted","Data":"acb622b0029198bcbc18982527576457e2a1b4b31c7318eaef5e26d850427508"}
Sep 18 19:39:14.460492 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:14.460446    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-6vdpn" podStartSLOduration=65.65496912 podStartE2EDuration="1m10.460432523s" totalImagesPullingTime="4.805463403s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:38:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:14.45932793 +0000 UTC m=+70.969684020" watchObservedRunningTime="2026-09-18 19:39:14.460432523 +0000 UTC m=+70.970788611"
Sep 18 19:39:14.473274 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:14.473233    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-857bd" podStartSLOduration=65.978609841 podStartE2EDuration="1m10.473219925s" totalImagesPullingTime="4.494610084s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:38:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:14.472279564 +0000 UTC m=+70.982635674" watchObservedRunningTime="2026-09-18 19:39:14.473219925 +0000 UTC m=+70.983576055"
Sep 18 19:39:14.489727 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:14.489677    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5f57555bf7-2b66f" podStartSLOduration=18.489662136 podStartE2EDuration="18.489662136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:38:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:14.488721971 +0000 UTC m=+70.999078078" watchObservedRunningTime="2026-09-18 19:39:14.489662136 +0000 UTC m=+71.000018226"
Sep 18 19:39:14.578904 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:14.578822    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:14.679712 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:14.679675    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:15.179492 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:15.179454    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:15.780397 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:15.780360    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:15.919266 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:15.919237    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:16.019100 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:16.019062    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:16.171701 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:16.171667    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:16.379619 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:16.379586    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:16.459300 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:16.459216    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-w4mzh" event={"ID":"18f89927-0873-43f7-b237-ec56aaacb7c0","Type":"ContainerStarted","Data":"19e0a6ceedd728afb888d97cdbda7a1d19f10b06f90b9d4db08c1b1bd59f700f"}
Sep 18 19:39:16.459300 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:16.459277    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-w4mzh" event={"ID":"18f89927-0873-43f7-b237-ec56aaacb7c0","Type":"ContainerStarted","Data":"8929436fffe64f38b4cfc0a6e373f9b5d1936b9fdc498814f229c0929811cc1d"}
Sep 18 19:39:16.473879 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:16.473825    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-w4mzh" podStartSLOduration=18.888997371 podStartE2EDuration="20.47380812s" totalImagesPullingTime="1.584810749s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:38:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:16.473448611 +0000 UTC m=+72.983804704" watchObservedRunningTime="2026-09-18 19:39:16.47380812 +0000 UTC m=+72.984164212"
Sep 18 19:39:17.176381 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:17.176343    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:17.788652 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:17.788619    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:17.882194 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:17.882139    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2c9bbaf8-3c60-402a-a13f-03d8e012c7c1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jfgj8\" (UID: \"2c9bbaf8-3c60-402a-a13f-03d8e012c7c1\") " pod="openshift-insights/insights-runtime-extractor-jfgj8"
Sep 18 19:39:17.882194 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:17.882199    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2c9bbaf8-3c60-402a-a13f-03d8e012c7c1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jfgj8\" (UID: \"2c9bbaf8-3c60-402a-a13f-03d8e012c7c1\") " pod="openshift-insights/insights-runtime-extractor-jfgj8"
Sep 18 19:39:17.882387 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:39:17.882319    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:39:17.882387 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:39:17.882373    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c9bbaf8-3c60-402a-a13f-03d8e012c7c1-insights-runtime-extractor-tls podName:2c9bbaf8-3c60-402a-a13f-03d8e012c7c1 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:33.882359598 +0000 UTC m=+90.392715681 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2c9bbaf8-3c60-402a-a13f-03d8e012c7c1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jfgj8" (UID: "2c9bbaf8-3c60-402a-a13f-03d8e012c7c1") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:39:17.882698 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:17.882680    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2c9bbaf8-3c60-402a-a13f-03d8e012c7c1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jfgj8\" (UID: \"2c9bbaf8-3c60-402a-a13f-03d8e012c7c1\") " pod="openshift-insights/insights-runtime-extractor-jfgj8"
Sep 18 19:39:17.975753 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:17.975718    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:18.186103 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:18.186060    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:18.585724 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:18.585635    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:18.778599 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:18.778569    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:18.994237 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:18.994207    2564 scope.go:117] "RemoveContainer" containerID="8600f4eb75644be5811b55bf1c8d09e85718e26301bd7d3ee3ce0e5fedfbf263"
Sep 18 19:39:19.471523 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:19.471492    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-58vl4_945a444e-9381-4e2f-b4c1-64be0ec88e6c/console-operator/2.log"
Sep 18 19:39:19.472015 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:19.471976    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-58vl4_945a444e-9381-4e2f-b4c1-64be0ec88e6c/console-operator/1.log"
Sep 18 19:39:19.472078 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:19.472034    2564 generic.go:356] "Generic (PLEG): container finished" podID="945a444e-9381-4e2f-b4c1-64be0ec88e6c" containerID="7469e6446d62a75aee2a27153ed14231911828be2895ba72f096a975ec76fa9e" exitCode=255
Sep 18 19:39:19.472078 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:19.472063    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-58vl4" event={"ID":"945a444e-9381-4e2f-b4c1-64be0ec88e6c","Type":"ContainerDied","Data":"7469e6446d62a75aee2a27153ed14231911828be2895ba72f096a975ec76fa9e"}
Sep 18 19:39:19.472142 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:19.472106    2564 scope.go:117] "RemoveContainer" containerID="8600f4eb75644be5811b55bf1c8d09e85718e26301bd7d3ee3ce0e5fedfbf263"
Sep 18 19:39:19.472508 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:19.472492    2564 scope.go:117] "RemoveContainer" containerID="7469e6446d62a75aee2a27153ed14231911828be2895ba72f096a975ec76fa9e"
Sep 18 19:39:19.472671 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:39:19.472652    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-fd4bb46c-58vl4_openshift-console-operator(945a444e-9381-4e2f-b4c1-64be0ec88e6c)\"" pod="openshift-console-operator/console-operator-fd4bb46c-58vl4" podUID="945a444e-9381-4e2f-b4c1-64be0ec88e6c"
Sep 18 19:39:20.476394 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:20.476368    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-58vl4_945a444e-9381-4e2f-b4c1-64be0ec88e6c/console-operator/2.log"
Sep 18 19:39:26.199175 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:26.199056    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5f57555bf7-2b66f"]
Sep 18 19:39:26.239124 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:26.239088    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-67dc7fd598-rjlqj"]
Sep 18 19:39:26.243293 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:26.243270    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67dc7fd598-rjlqj"
Sep 18 19:39:26.258799 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:26.258766    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-67dc7fd598-rjlqj"]
Sep 18 19:39:26.334158 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:26.334122    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:26.355660 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:26.355623    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f280ba4e-c665-4351-8450-144134ea6c8b-bound-sa-token\") pod \"image-registry-67dc7fd598-rjlqj\" (UID: \"f280ba4e-c665-4351-8450-144134ea6c8b\") " pod="openshift-image-registry/image-registry-67dc7fd598-rjlqj"
Sep 18 19:39:26.355660 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:26.355663    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f280ba4e-c665-4351-8450-144134ea6c8b-installation-pull-secrets\") pod \"image-registry-67dc7fd598-rjlqj\" (UID: \"f280ba4e-c665-4351-8450-144134ea6c8b\") " pod="openshift-image-registry/image-registry-67dc7fd598-rjlqj"
Sep 18 19:39:26.355873 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:26.355756    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f280ba4e-c665-4351-8450-144134ea6c8b-ca-trust-extracted\") pod \"image-registry-67dc7fd598-rjlqj\" (UID: \"f280ba4e-c665-4351-8450-144134ea6c8b\") " pod="openshift-image-registry/image-registry-67dc7fd598-rjlqj"
Sep 18 19:39:26.355873 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:26.355794    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqjss\" (UniqueName: \"kubernetes.io/projected/f280ba4e-c665-4351-8450-144134ea6c8b-kube-api-access-cqjss\") pod \"image-registry-67dc7fd598-rjlqj\" (UID: \"f280ba4e-c665-4351-8450-144134ea6c8b\") " pod="openshift-image-registry/image-registry-67dc7fd598-rjlqj"
Sep 18 19:39:26.355873 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:26.355826    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f280ba4e-c665-4351-8450-144134ea6c8b-registry-tls\") pod \"image-registry-67dc7fd598-rjlqj\" (UID: \"f280ba4e-c665-4351-8450-144134ea6c8b\") " pod="openshift-image-registry/image-registry-67dc7fd598-rjlqj"
Sep 18 19:39:26.356005 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:26.355881    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f280ba4e-c665-4351-8450-144134ea6c8b-image-registry-private-configuration\") pod \"image-registry-67dc7fd598-rjlqj\" (UID: \"f280ba4e-c665-4351-8450-144134ea6c8b\") " pod="openshift-image-registry/image-registry-67dc7fd598-rjlqj"
Sep 18 19:39:26.356005 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:26.355912    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f280ba4e-c665-4351-8450-144134ea6c8b-trusted-ca\") pod \"image-registry-67dc7fd598-rjlqj\" (UID: \"f280ba4e-c665-4351-8450-144134ea6c8b\") " pod="openshift-image-registry/image-registry-67dc7fd598-rjlqj"
Sep 18 19:39:26.356005 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:26.355943    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f280ba4e-c665-4351-8450-144134ea6c8b-registry-certificates\") pod \"image-registry-67dc7fd598-rjlqj\" (UID: \"f280ba4e-c665-4351-8450-144134ea6c8b\") " pod="openshift-image-registry/image-registry-67dc7fd598-rjlqj"
Sep 18 19:39:26.434131 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:26.434085    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:26.457412 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:26.457313    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f280ba4e-c665-4351-8450-144134ea6c8b-installation-pull-secrets\") pod \"image-registry-67dc7fd598-rjlqj\" (UID: \"f280ba4e-c665-4351-8450-144134ea6c8b\") " pod="openshift-image-registry/image-registry-67dc7fd598-rjlqj"
Sep 18 19:39:26.457412 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:26.457377    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f280ba4e-c665-4351-8450-144134ea6c8b-ca-trust-extracted\") pod \"image-registry-67dc7fd598-rjlqj\" (UID: \"f280ba4e-c665-4351-8450-144134ea6c8b\") " pod="openshift-image-registry/image-registry-67dc7fd598-rjlqj"
Sep 18 19:39:26.457412 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:26.457411    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqjss\" (UniqueName: \"kubernetes.io/projected/f280ba4e-c665-4351-8450-144134ea6c8b-kube-api-access-cqjss\") pod \"image-registry-67dc7fd598-rjlqj\" (UID: \"f280ba4e-c665-4351-8450-144134ea6c8b\") " pod="openshift-image-registry/image-registry-67dc7fd598-rjlqj"
Sep 18 19:39:26.457683 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:26.457448    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f280ba4e-c665-4351-8450-144134ea6c8b-registry-tls\") pod \"image-registry-67dc7fd598-rjlqj\" (UID: \"f280ba4e-c665-4351-8450-144134ea6c8b\") " pod="openshift-image-registry/image-registry-67dc7fd598-rjlqj"
Sep 18 19:39:26.457683 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:26.457472    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f280ba4e-c665-4351-8450-144134ea6c8b-image-registry-private-configuration\") pod \"image-registry-67dc7fd598-rjlqj\" (UID: \"f280ba4e-c665-4351-8450-144134ea6c8b\") " pod="openshift-image-registry/image-registry-67dc7fd598-rjlqj"
Sep 18 19:39:26.457683 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:26.457496    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f280ba4e-c665-4351-8450-144134ea6c8b-trusted-ca\") pod \"image-registry-67dc7fd598-rjlqj\" (UID: \"f280ba4e-c665-4351-8450-144134ea6c8b\") " pod="openshift-image-registry/image-registry-67dc7fd598-rjlqj"
Sep 18 19:39:26.457683 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:26.457539    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f280ba4e-c665-4351-8450-144134ea6c8b-registry-certificates\") pod \"image-registry-67dc7fd598-rjlqj\" (UID: \"f280ba4e-c665-4351-8450-144134ea6c8b\") " pod="openshift-image-registry/image-registry-67dc7fd598-rjlqj"
Sep 18 19:39:26.457683 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:26.457639    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f280ba4e-c665-4351-8450-144134ea6c8b-bound-sa-token\") pod \"image-registry-67dc7fd598-rjlqj\" (UID: \"f280ba4e-c665-4351-8450-144134ea6c8b\") " pod="openshift-image-registry/image-registry-67dc7fd598-rjlqj"
Sep 18 19:39:26.457915 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:26.457879    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f280ba4e-c665-4351-8450-144134ea6c8b-ca-trust-extracted\") pod \"image-registry-67dc7fd598-rjlqj\" (UID: \"f280ba4e-c665-4351-8450-144134ea6c8b\") " pod="openshift-image-registry/image-registry-67dc7fd598-rjlqj"
Sep 18 19:39:26.458509 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:26.458483    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f280ba4e-c665-4351-8450-144134ea6c8b-registry-certificates\") pod \"image-registry-67dc7fd598-rjlqj\" (UID: \"f280ba4e-c665-4351-8450-144134ea6c8b\") " pod="openshift-image-registry/image-registry-67dc7fd598-rjlqj"
Sep 18 19:39:26.458666 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:26.458649    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f280ba4e-c665-4351-8450-144134ea6c8b-trusted-ca\") pod \"image-registry-67dc7fd598-rjlqj\" (UID: \"f280ba4e-c665-4351-8450-144134ea6c8b\") " pod="openshift-image-registry/image-registry-67dc7fd598-rjlqj"
Sep 18 19:39:26.460093 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:26.460072    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f280ba4e-c665-4351-8450-144134ea6c8b-registry-tls\") pod \"image-registry-67dc7fd598-rjlqj\" (UID: \"f280ba4e-c665-4351-8450-144134ea6c8b\") " pod="openshift-image-registry/image-registry-67dc7fd598-rjlqj"
Sep 18 19:39:26.460418 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:26.460397    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f280ba4e-c665-4351-8450-144134ea6c8b-image-registry-private-configuration\") pod \"image-registry-67dc7fd598-rjlqj\" (UID: \"f280ba4e-c665-4351-8450-144134ea6c8b\") " pod="openshift-image-registry/image-registry-67dc7fd598-rjlqj"
Sep 18 19:39:26.460486 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:26.460444    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f280ba4e-c665-4351-8450-144134ea6c8b-installation-pull-secrets\") pod \"image-registry-67dc7fd598-rjlqj\" (UID: \"f280ba4e-c665-4351-8450-144134ea6c8b\") " pod="openshift-image-registry/image-registry-67dc7fd598-rjlqj"
Sep 18 19:39:26.466023 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:26.465999    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqjss\" (UniqueName: \"kubernetes.io/projected/f280ba4e-c665-4351-8450-144134ea6c8b-kube-api-access-cqjss\") pod \"image-registry-67dc7fd598-rjlqj\" (UID: \"f280ba4e-c665-4351-8450-144134ea6c8b\") " pod="openshift-image-registry/image-registry-67dc7fd598-rjlqj"
Sep 18 19:39:26.466150 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:26.466037    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f280ba4e-c665-4351-8450-144134ea6c8b-bound-sa-token\") pod \"image-registry-67dc7fd598-rjlqj\" (UID: \"f280ba4e-c665-4351-8450-144134ea6c8b\") " pod="openshift-image-registry/image-registry-67dc7fd598-rjlqj"
Sep 18 19:39:26.552154 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:26.552113    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67dc7fd598-rjlqj"
Sep 18 19:39:26.825080 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:26.825003    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-67dc7fd598-rjlqj"]
Sep 18 19:39:26.828072 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:39:26.828039    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf280ba4e_c665_4351_8450_144134ea6c8b.slice/crio-2c498c4f17114ceec77d71e976c8319da4caa7278afe11aa4f55706ff674cf23 WatchSource:0}: Error finding container 2c498c4f17114ceec77d71e976c8319da4caa7278afe11aa4f55706ff674cf23: Status 404 returned error can't find the container with id 2c498c4f17114ceec77d71e976c8319da4caa7278afe11aa4f55706ff674cf23
Sep 18 19:39:27.054420 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:27.054360    2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-58vl4"
Sep 18 19:39:27.054611 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:27.054447    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-58vl4"
Sep 18 19:39:27.054802 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:27.054786    2564 scope.go:117] "RemoveContainer" containerID="7469e6446d62a75aee2a27153ed14231911828be2895ba72f096a975ec76fa9e"
Sep 18 19:39:27.054975 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:39:27.054954    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-fd4bb46c-58vl4_openshift-console-operator(945a444e-9381-4e2f-b4c1-64be0ec88e6c)\"" pod="openshift-console-operator/console-operator-fd4bb46c-58vl4" podUID="945a444e-9381-4e2f-b4c1-64be0ec88e6c"
Sep 18 19:39:27.500039 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:27.500005    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67dc7fd598-rjlqj" event={"ID":"f280ba4e-c665-4351-8450-144134ea6c8b","Type":"ContainerStarted","Data":"0311f13754a5b2a55a4fb7b61dd599c2b3aceca32b581d4c40f2c82a7bd28233"}
Sep 18 19:39:27.500442 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:27.500052    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67dc7fd598-rjlqj" event={"ID":"f280ba4e-c665-4351-8450-144134ea6c8b","Type":"ContainerStarted","Data":"2c498c4f17114ceec77d71e976c8319da4caa7278afe11aa4f55706ff674cf23"}
Sep 18 19:39:27.500442 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:27.500125    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-67dc7fd598-rjlqj"
Sep 18 19:39:27.518634 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:27.518576    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-67dc7fd598-rjlqj" podStartSLOduration=1.518557647 podStartE2EDuration="1.518557647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:39:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:27.517301031 +0000 UTC m=+84.027657119" watchObservedRunningTime="2026-09-18 19:39:27.518557647 +0000 UTC m=+84.028913779"
Sep 18 19:39:28.576675 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:28.576633    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20027a0c-0095-4418-aaa4-d0a6f273ef85-metrics-certs\") pod \"router-default-756fcb9d6-47xpw\" (UID: \"20027a0c-0095-4418-aaa4-d0a6f273ef85\") " pod="openshift-ingress/router-default-756fcb9d6-47xpw"
Sep 18 19:39:28.576675 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:28.576679    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20027a0c-0095-4418-aaa4-d0a6f273ef85-service-ca-bundle\") pod \"router-default-756fcb9d6-47xpw\" (UID: \"20027a0c-0095-4418-aaa4-d0a6f273ef85\") " pod="openshift-ingress/router-default-756fcb9d6-47xpw"
Sep 18 19:39:28.577309 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:28.577292    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20027a0c-0095-4418-aaa4-d0a6f273ef85-service-ca-bundle\") pod \"router-default-756fcb9d6-47xpw\" (UID: \"20027a0c-0095-4418-aaa4-d0a6f273ef85\") " pod="openshift-ingress/router-default-756fcb9d6-47xpw"
Sep 18 19:39:28.579182 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:28.579158    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20027a0c-0095-4418-aaa4-d0a6f273ef85-metrics-certs\") pod \"router-default-756fcb9d6-47xpw\" (UID: \"20027a0c-0095-4418-aaa4-d0a6f273ef85\") " pod="openshift-ingress/router-default-756fcb9d6-47xpw"
Sep 18 19:39:28.770416 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:28.770387    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-9lg52\""
Sep 18 19:39:28.778277 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:28.778252    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-756fcb9d6-47xpw"
Sep 18 19:39:29.052486 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:29.052453    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-756fcb9d6-47xpw"]
Sep 18 19:39:29.055149 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:39:29.055121    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20027a0c_0095_4418_aaa4_d0a6f273ef85.slice/crio-2b8ff8a5ef19db6a8788a218f9527e1345b42127dc9c23bdafeb7d7e0cb8da87 WatchSource:0}: Error finding container 2b8ff8a5ef19db6a8788a218f9527e1345b42127dc9c23bdafeb7d7e0cb8da87: Status 404 returned error can't find the container with id 2b8ff8a5ef19db6a8788a218f9527e1345b42127dc9c23bdafeb7d7e0cb8da87
Sep 18 19:39:29.381190 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:29.381161    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:29.481997 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:29.481948    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:29.508892 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:29.508859    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-756fcb9d6-47xpw" event={"ID":"20027a0c-0095-4418-aaa4-d0a6f273ef85","Type":"ContainerStarted","Data":"175479fbb6b018f02bb8d48f9cb711c7935578ee05474605c11f6de79528c682"}
Sep 18 19:39:29.509075 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:29.508910    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-756fcb9d6-47xpw" event={"ID":"20027a0c-0095-4418-aaa4-d0a6f273ef85","Type":"ContainerStarted","Data":"2b8ff8a5ef19db6a8788a218f9527e1345b42127dc9c23bdafeb7d7e0cb8da87"}
Sep 18 19:39:29.527476 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:29.527421    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-756fcb9d6-47xpw" podStartSLOduration=33.527401021 podStartE2EDuration="33.527401021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:38:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:29.526512212 +0000 UTC m=+86.036868315" watchObservedRunningTime="2026-09-18 19:39:29.527401021 +0000 UTC m=+86.037757111"
Sep 18 19:39:29.688418 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:29.688299    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:29.747129 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:29.747092    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:29.778488 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:29.778454    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-756fcb9d6-47xpw"
Sep 18 19:39:29.781095 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:29.781069    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-756fcb9d6-47xpw"
Sep 18 19:39:30.512840 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:30.512806    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-756fcb9d6-47xpw"
Sep 18 19:39:30.514245 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:30.514222    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-756fcb9d6-47xpw"
Sep 18 19:39:32.037704 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:32.037667    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:32.097957 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:32.097919    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:33.230976 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:33.230942    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:33.333033 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:33.332906    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:33.390314 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:33.390274    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-xw6cl"]
Sep 18 19:39:33.394563 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:33.394538    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-xw6cl"
Sep 18 19:39:33.396619 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:33.396597    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 18 19:39:33.396760 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:33.396630    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-b8tnb\""
Sep 18 19:39:33.400729 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:33.400703    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-xw6cl"]
Sep 18 19:39:33.405866 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:33.405838    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:33.512759 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:33.512678    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c0f6cc20-e321-47c5-a2b0-ea3d9d29cbe5-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-xw6cl\" (UID: \"c0f6cc20-e321-47c5-a2b0-ea3d9d29cbe5\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-xw6cl"
Sep 18 19:39:33.613429 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:33.613386    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c0f6cc20-e321-47c5-a2b0-ea3d9d29cbe5-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-xw6cl\" (UID: \"c0f6cc20-e321-47c5-a2b0-ea3d9d29cbe5\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-xw6cl"
Sep 18 19:39:33.613621 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:39:33.613544    2564 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 18 19:39:33.613621 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:39:33.613610    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0f6cc20-e321-47c5-a2b0-ea3d9d29cbe5-tls-certificates podName:c0f6cc20-e321-47c5-a2b0-ea3d9d29cbe5 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:34.113593438 +0000 UTC m=+90.623949520 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/c0f6cc20-e321-47c5-a2b0-ea3d9d29cbe5-tls-certificates") pod "prometheus-operator-admission-webhook-5c6747dc7b-xw6cl" (UID: "c0f6cc20-e321-47c5-a2b0-ea3d9d29cbe5") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 18 19:39:33.718708 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:33.718672    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:33.915997 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:33.915953    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2c9bbaf8-3c60-402a-a13f-03d8e012c7c1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jfgj8\" (UID: \"2c9bbaf8-3c60-402a-a13f-03d8e012c7c1\") " pod="openshift-insights/insights-runtime-extractor-jfgj8"
Sep 18 19:39:33.918348 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:33.918319    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2c9bbaf8-3c60-402a-a13f-03d8e012c7c1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jfgj8\" (UID: \"2c9bbaf8-3c60-402a-a13f-03d8e012c7c1\") " pod="openshift-insights/insights-runtime-extractor-jfgj8"
Sep 18 19:39:34.117091 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:34.117056    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c0f6cc20-e321-47c5-a2b0-ea3d9d29cbe5-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-xw6cl\" (UID: \"c0f6cc20-e321-47c5-a2b0-ea3d9d29cbe5\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-xw6cl"
Sep 18 19:39:34.119507 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:34.119480    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c0f6cc20-e321-47c5-a2b0-ea3d9d29cbe5-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-xw6cl\" (UID: \"c0f6cc20-e321-47c5-a2b0-ea3d9d29cbe5\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-xw6cl"
Sep 18 19:39:34.164950 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:34.164921    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-696cc\""
Sep 18 19:39:34.173720 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:34.173649    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jfgj8"
Sep 18 19:39:34.303779 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:34.303746    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-xw6cl"
Sep 18 19:39:34.467603 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:34.467523    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jfgj8"]
Sep 18 19:39:34.470831 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:39:34.470804    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c9bbaf8_3c60_402a_a13f_03d8e012c7c1.slice/crio-76b245abf498f72e800f1f6ad1eaf91ae6d0a5df2447d7a64a88560321fb157b WatchSource:0}: Error finding container 76b245abf498f72e800f1f6ad1eaf91ae6d0a5df2447d7a64a88560321fb157b: Status 404 returned error can't find the container with id 76b245abf498f72e800f1f6ad1eaf91ae6d0a5df2447d7a64a88560321fb157b
Sep 18 19:39:34.526770 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:34.526743    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jfgj8" event={"ID":"2c9bbaf8-3c60-402a-a13f-03d8e012c7c1","Type":"ContainerStarted","Data":"7302ee134974e0ee25d0006d46b1fa9a4d00eaf97626cc7ecac4fc646f76dd26"}
Sep 18 19:39:34.526866 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:34.526823    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jfgj8" event={"ID":"2c9bbaf8-3c60-402a-a13f-03d8e012c7c1","Type":"ContainerStarted","Data":"76b245abf498f72e800f1f6ad1eaf91ae6d0a5df2447d7a64a88560321fb157b"}
Sep 18 19:39:34.590907 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:34.590875    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-xw6cl"]
Sep 18 19:39:34.594192 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:39:34.594165    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0f6cc20_e321_47c5_a2b0_ea3d9d29cbe5.slice/crio-08e0a2e752cfaed7fcac7981fa575eb3b5265334d243de8ceb94ae6c4a9d89a8 WatchSource:0}: Error finding container 08e0a2e752cfaed7fcac7981fa575eb3b5265334d243de8ceb94ae6c4a9d89a8: Status 404 returned error can't find the container with id 08e0a2e752cfaed7fcac7981fa575eb3b5265334d243de8ceb94ae6c4a9d89a8
Sep 18 19:39:35.532339 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:35.532245    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-xw6cl" event={"ID":"c0f6cc20-e321-47c5-a2b0-ea3d9d29cbe5","Type":"ContainerStarted","Data":"08e0a2e752cfaed7fcac7981fa575eb3b5265334d243de8ceb94ae6c4a9d89a8"}
Sep 18 19:39:35.534311 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:35.534279    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jfgj8" event={"ID":"2c9bbaf8-3c60-402a-a13f-03d8e012c7c1","Type":"ContainerStarted","Data":"4fa712e4c715895d161dc93577ed13998c13fae620cfaa9ab4bdc475b1e29ef3"}
Sep 18 19:39:36.205906 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:36.205875    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5f57555bf7-2b66f"
Sep 18 19:39:36.495820 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:36.495725    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:36.540539 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:36.540495    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-xw6cl" event={"ID":"c0f6cc20-e321-47c5-a2b0-ea3d9d29cbe5","Type":"ContainerStarted","Data":"7d0f30950b32360a78699d2603cfdf23b7779307332eabde7361a65fdabf89ca"}
Sep 18 19:39:36.541007 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:36.540932    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-xw6cl"
Sep 18 19:39:36.546134 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:36.546103    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-xw6cl"
Sep 18 19:39:36.555399 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:36.555344    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-xw6cl" podStartSLOduration=2.5323466 podStartE2EDuration="3.555322688s" totalImagesPullingTime="1.022976088s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:39:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:36.554287844 +0000 UTC m=+93.064643950" watchObservedRunningTime="2026-09-18 19:39:36.555322688 +0000 UTC m=+93.065678782"
Sep 18 19:39:36.558593 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:36.558564    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:37.416413 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:37.416376    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:37.462509 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:37.462476    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-zf82v"]
Sep 18 19:39:37.465544 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:37.465528    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-zf82v"
Sep 18 19:39:37.467804 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:37.467776    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 18 19:39:37.468423 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:37.468398    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 18 19:39:37.468534 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:37.468424    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 18 19:39:37.468534 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:37.468402    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 18 19:39:37.468534 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:37.468398    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 18 19:39:37.468534 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:37.468485    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-nv9s7\""
Sep 18 19:39:37.475290 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:37.475266    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-zf82v"]
Sep 18 19:39:37.489708 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:37.489678    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:37.546965 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:37.546929    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jfgj8" event={"ID":"2c9bbaf8-3c60-402a-a13f-03d8e012c7c1","Type":"ContainerStarted","Data":"6bad5fcfa38f9494ff0266c6bc79ccc3357834005f0d5d1382dd02220221afcb"}
Sep 18 19:39:37.565903 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:37.565417    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-jfgj8" podStartSLOduration=33.360315393 podStartE2EDuration="35.565396307s" totalImagesPullingTime="2.205080914s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:39:02 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:37.564201254 +0000 UTC m=+94.074557356" watchObservedRunningTime="2026-09-18 19:39:37.565396307 +0000 UTC m=+94.075752401"
Sep 18 19:39:37.647377 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:37.647330    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt8xg\" (UniqueName: \"kubernetes.io/projected/e9a41502-dda4-4946-8ef5-bf9c8e5035b9-kube-api-access-wt8xg\") pod \"prometheus-operator-7bd5d56d6-zf82v\" (UID: \"e9a41502-dda4-4946-8ef5-bf9c8e5035b9\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-zf82v"
Sep 18 19:39:37.647377 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:37.647374    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e9a41502-dda4-4946-8ef5-bf9c8e5035b9-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-zf82v\" (UID: \"e9a41502-dda4-4946-8ef5-bf9c8e5035b9\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-zf82v"
Sep 18 19:39:37.647588 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:37.647437    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e9a41502-dda4-4946-8ef5-bf9c8e5035b9-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-zf82v\" (UID: \"e9a41502-dda4-4946-8ef5-bf9c8e5035b9\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-zf82v"
Sep 18 19:39:37.647588 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:37.647474    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e9a41502-dda4-4946-8ef5-bf9c8e5035b9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-zf82v\" (UID: \"e9a41502-dda4-4946-8ef5-bf9c8e5035b9\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-zf82v"
Sep 18 19:39:37.719101 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:37.719009    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:37.748528 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:37.748492    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e9a41502-dda4-4946-8ef5-bf9c8e5035b9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-zf82v\" (UID: \"e9a41502-dda4-4946-8ef5-bf9c8e5035b9\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-zf82v"
Sep 18 19:39:37.748632 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:37.748597    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt8xg\" (UniqueName: \"kubernetes.io/projected/e9a41502-dda4-4946-8ef5-bf9c8e5035b9-kube-api-access-wt8xg\") pod \"prometheus-operator-7bd5d56d6-zf82v\" (UID: \"e9a41502-dda4-4946-8ef5-bf9c8e5035b9\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-zf82v"
Sep 18 19:39:37.748669 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:37.748629    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e9a41502-dda4-4946-8ef5-bf9c8e5035b9-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-zf82v\" (UID: \"e9a41502-dda4-4946-8ef5-bf9c8e5035b9\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-zf82v"
Sep 18 19:39:37.748706 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:37.748671    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e9a41502-dda4-4946-8ef5-bf9c8e5035b9-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-zf82v\" (UID: \"e9a41502-dda4-4946-8ef5-bf9c8e5035b9\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-zf82v"
Sep 18 19:39:37.749425 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:37.749399    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e9a41502-dda4-4946-8ef5-bf9c8e5035b9-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-zf82v\" (UID: \"e9a41502-dda4-4946-8ef5-bf9c8e5035b9\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-zf82v"
Sep 18 19:39:37.750969 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:37.750950    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e9a41502-dda4-4946-8ef5-bf9c8e5035b9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-zf82v\" (UID: \"e9a41502-dda4-4946-8ef5-bf9c8e5035b9\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-zf82v"
Sep 18 19:39:37.751092 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:37.751049    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e9a41502-dda4-4946-8ef5-bf9c8e5035b9-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-zf82v\" (UID: \"e9a41502-dda4-4946-8ef5-bf9c8e5035b9\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-zf82v"
Sep 18 19:39:37.756827 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:37.756803    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt8xg\" (UniqueName: \"kubernetes.io/projected/e9a41502-dda4-4946-8ef5-bf9c8e5035b9-kube-api-access-wt8xg\") pod \"prometheus-operator-7bd5d56d6-zf82v\" (UID: \"e9a41502-dda4-4946-8ef5-bf9c8e5035b9\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-zf82v"
Sep 18 19:39:37.774785 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:37.774753    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-zf82v"
Sep 18 19:39:38.047559 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:38.047172    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-zf82v"]
Sep 18 19:39:38.049855 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:39:38.049827    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9a41502_dda4_4946_8ef5_bf9c8e5035b9.slice/crio-3e380f946226b4edeae1791c71b85081cb5917184f00aea4d471858d0313f97c WatchSource:0}: Error finding container 3e380f946226b4edeae1791c71b85081cb5917184f00aea4d471858d0313f97c: Status 404 returned error can't find the container with id 3e380f946226b4edeae1791c71b85081cb5917184f00aea4d471858d0313f97c
Sep 18 19:39:38.551398 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:38.551364    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-zf82v" event={"ID":"e9a41502-dda4-4946-8ef5-bf9c8e5035b9","Type":"ContainerStarted","Data":"3e380f946226b4edeae1791c71b85081cb5917184f00aea4d471858d0313f97c"}
Sep 18 19:39:38.994181 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:38.994150    2564 scope.go:117] "RemoveContainer" containerID="7469e6446d62a75aee2a27153ed14231911828be2895ba72f096a975ec76fa9e"
Sep 18 19:39:38.994393 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:39:38.994370    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-fd4bb46c-58vl4_openshift-console-operator(945a444e-9381-4e2f-b4c1-64be0ec88e6c)\"" pod="openshift-console-operator/console-operator-fd4bb46c-58vl4" podUID="945a444e-9381-4e2f-b4c1-64be0ec88e6c"
Sep 18 19:39:39.246552 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:39.246464    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:39.310508 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:39.310465    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:39.479080 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:39.479045    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:39.557591 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:39.557508    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-zf82v" event={"ID":"e9a41502-dda4-4946-8ef5-bf9c8e5035b9","Type":"ContainerStarted","Data":"1e8c65e4a51a266b8120eaad019dcce4d3d7db635df3fee092e1a2b203f2a8f3"}
Sep 18 19:39:39.557591 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:39.557560    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-zf82v" event={"ID":"e9a41502-dda4-4946-8ef5-bf9c8e5035b9","Type":"ContainerStarted","Data":"f4142985a5756939a039b884f69dd9b3218622130c8a7e459c672ceca5f2ca42"}
Sep 18 19:39:39.573717 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:39.573647    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-zf82v" podStartSLOduration=1.252020868 podStartE2EDuration="2.573622642s" totalImagesPullingTime="1.321601774s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:39:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:39.572442466 +0000 UTC m=+96.082798770" watchObservedRunningTime="2026-09-18 19:39:39.573622642 +0000 UTC m=+96.083978732"
Sep 18 19:39:39.577812 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:39.577775    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:39.645599 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:39.645563    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:41.578831 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:41.578786    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:41.676341 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:41.676311    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:41.789720 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:41.789689    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:41.805823 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:41.805768    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-mxhbj"]
Sep 18 19:39:41.809397 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:41.809374    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-mxhbj"
Sep 18 19:39:41.811949 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:41.811924    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-lkjwj\""
Sep 18 19:39:41.812084 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:41.811949    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 18 19:39:41.812084 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:41.812026    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 18 19:39:41.816155 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:41.816086    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-2ls7m"]
Sep 18 19:39:41.819140 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:41.819122    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2ls7m"
Sep 18 19:39:41.820953 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:41.820925    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 18 19:39:41.821394 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:41.821367    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 18 19:39:41.821680 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:41.821663    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 18 19:39:41.821764 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:41.821666    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-qmpz8\""
Sep 18 19:39:41.821924 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:41.821906    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-mxhbj"]
Sep 18 19:39:41.832117 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:41.832051    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-n86fq"]
Sep 18 19:39:41.835337 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:41.835316    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-n86fq"
Sep 18 19:39:41.837182 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:41.837147    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 18 19:39:41.837301 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:41.837183    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 18 19:39:41.837573 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:41.837555    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 18 19:39:41.837662 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:41.837563    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-jsfbh\""
Sep 18 19:39:41.839862 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:41.839839    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-n86fq"]
Sep 18 19:39:41.858581 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:41.858548    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:41.880874 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:41.880839    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8b6ee93c-e19d-44f2-9ba9-ae133edb7686-metrics-tls\") pod \"dns-default-5krlr\" (UID: \"8b6ee93c-e19d-44f2-9ba9-ae133edb7686\") " pod="openshift-dns/dns-default-5krlr"
Sep 18 19:39:41.883127 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:41.883106    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8b6ee93c-e19d-44f2-9ba9-ae133edb7686-metrics-tls\") pod \"dns-default-5krlr\" (UID: \"8b6ee93c-e19d-44f2-9ba9-ae133edb7686\") " pod="openshift-dns/dns-default-5krlr"
Sep 18 19:39:41.923170 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:41.923137    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:41.981440 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:41.981402    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2ee2e08b-c234-4675-83ba-a76063ea1285-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-n86fq\" (UID: \"2ee2e08b-c234-4675-83ba-a76063ea1285\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-n86fq"
Sep 18 19:39:41.981440 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:41.981440    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d5526936-3945-4732-bb5e-515d905fa913-node-exporter-textfile\") pod \"node-exporter-2ls7m\" (UID: \"d5526936-3945-4732-bb5e-515d905fa913\") " pod="openshift-monitoring/node-exporter-2ls7m"
Sep 18 19:39:41.981629 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:41.981464    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d5526936-3945-4732-bb5e-515d905fa913-node-exporter-wtmp\") pod \"node-exporter-2ls7m\" (UID: \"d5526936-3945-4732-bb5e-515d905fa913\") " pod="openshift-monitoring/node-exporter-2ls7m"
Sep 18 19:39:41.981629 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:41.981490    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d5526936-3945-4732-bb5e-515d905fa913-node-exporter-accelerators-collector-config\") pod \"node-exporter-2ls7m\" (UID: \"d5526936-3945-4732-bb5e-515d905fa913\") " pod="openshift-monitoring/node-exporter-2ls7m"
Sep 18 19:39:41.981629 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:41.981516    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d5526936-3945-4732-bb5e-515d905fa913-node-exporter-tls\") pod \"node-exporter-2ls7m\" (UID: \"d5526936-3945-4732-bb5e-515d905fa913\") " pod="openshift-monitoring/node-exporter-2ls7m"
Sep 18 19:39:41.981629 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:41.981547    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78bfv\" (UniqueName: \"kubernetes.io/projected/d5526936-3945-4732-bb5e-515d905fa913-kube-api-access-78bfv\") pod \"node-exporter-2ls7m\" (UID: \"d5526936-3945-4732-bb5e-515d905fa913\") " pod="openshift-monitoring/node-exporter-2ls7m"
Sep 18 19:39:41.981629 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:41.981592    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25c36b4f-3ee8-48f3-83aa-b816203e1335-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-mxhbj\" (UID: \"25c36b4f-3ee8-48f3-83aa-b816203e1335\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-mxhbj"
Sep 18 19:39:41.981629 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:41.981606    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5526936-3945-4732-bb5e-515d905fa913-metrics-client-ca\") pod \"node-exporter-2ls7m\" (UID: \"d5526936-3945-4732-bb5e-515d905fa913\") " pod="openshift-monitoring/node-exporter-2ls7m"
Sep 18 19:39:41.981629 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:41.981622    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d5526936-3945-4732-bb5e-515d905fa913-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2ls7m\" (UID: \"d5526936-3945-4732-bb5e-515d905fa913\") " pod="openshift-monitoring/node-exporter-2ls7m"
Sep 18 19:39:41.981861 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:41.981682    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/25c36b4f-3ee8-48f3-83aa-b816203e1335-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-mxhbj\" (UID: \"25c36b4f-3ee8-48f3-83aa-b816203e1335\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-mxhbj"
Sep 18 19:39:41.981861 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:41.981711    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ee2e08b-c234-4675-83ba-a76063ea1285-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-n86fq\" (UID: \"2ee2e08b-c234-4675-83ba-a76063ea1285\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-n86fq"
Sep 18 19:39:41.981861 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:41.981728    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d5526936-3945-4732-bb5e-515d905fa913-sys\") pod \"node-exporter-2ls7m\" (UID: \"d5526936-3945-4732-bb5e-515d905fa913\") " pod="openshift-monitoring/node-exporter-2ls7m"
Sep 18 19:39:41.981861 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:41.981748    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/25c36b4f-3ee8-48f3-83aa-b816203e1335-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-mxhbj\" (UID: \"25c36b4f-3ee8-48f3-83aa-b816203e1335\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-mxhbj"
Sep 18 19:39:41.981861 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:41.981769    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2ee2e08b-c234-4675-83ba-a76063ea1285-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-n86fq\" (UID: \"2ee2e08b-c234-4675-83ba-a76063ea1285\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-n86fq"
Sep 18 19:39:41.981861 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:41.981786    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2ee2e08b-c234-4675-83ba-a76063ea1285-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-n86fq\" (UID: \"2ee2e08b-c234-4675-83ba-a76063ea1285\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-n86fq"
Sep 18 19:39:41.981861 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:41.981804    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2ee2e08b-c234-4675-83ba-a76063ea1285-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-n86fq\" (UID: \"2ee2e08b-c234-4675-83ba-a76063ea1285\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-n86fq"
Sep 18 19:39:41.981861 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:41.981820    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d5526936-3945-4732-bb5e-515d905fa913-root\") pod \"node-exporter-2ls7m\" (UID: \"d5526936-3945-4732-bb5e-515d905fa913\") " pod="openshift-monitoring/node-exporter-2ls7m"
Sep 18 19:39:41.981861 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:41.981841    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5tqh\" (UniqueName: \"kubernetes.io/projected/2ee2e08b-c234-4675-83ba-a76063ea1285-kube-api-access-g5tqh\") pod \"kube-state-metrics-84c885658c-n86fq\" (UID: \"2ee2e08b-c234-4675-83ba-a76063ea1285\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-n86fq"
Sep 18 19:39:41.982191 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:41.981911    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t652n\" (UniqueName: \"kubernetes.io/projected/25c36b4f-3ee8-48f3-83aa-b816203e1335-kube-api-access-t652n\") pod \"openshift-state-metrics-86c5bc79d6-mxhbj\" (UID: \"25c36b4f-3ee8-48f3-83aa-b816203e1335\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-mxhbj"
Sep 18 19:39:42.058810 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:42.058777    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:42.082881 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:42.082803    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25c36b4f-3ee8-48f3-83aa-b816203e1335-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-mxhbj\" (UID: \"25c36b4f-3ee8-48f3-83aa-b816203e1335\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-mxhbj"
Sep 18 19:39:42.082881 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:42.082837    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5526936-3945-4732-bb5e-515d905fa913-metrics-client-ca\") pod \"node-exporter-2ls7m\" (UID: \"d5526936-3945-4732-bb5e-515d905fa913\") " pod="openshift-monitoring/node-exporter-2ls7m"
Sep 18 19:39:42.082881 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:42.082853    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d5526936-3945-4732-bb5e-515d905fa913-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2ls7m\" (UID: \"d5526936-3945-4732-bb5e-515d905fa913\") " pod="openshift-monitoring/node-exporter-2ls7m"
Sep 18 19:39:42.082881 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:42.082872    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/25c36b4f-3ee8-48f3-83aa-b816203e1335-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-mxhbj\" (UID: \"25c36b4f-3ee8-48f3-83aa-b816203e1335\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-mxhbj"
Sep 18 19:39:42.083216 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:42.082899    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ee2e08b-c234-4675-83ba-a76063ea1285-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-n86fq\" (UID: \"2ee2e08b-c234-4675-83ba-a76063ea1285\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-n86fq"
Sep 18 19:39:42.083216 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:42.082920    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d5526936-3945-4732-bb5e-515d905fa913-sys\") pod \"node-exporter-2ls7m\" (UID: \"d5526936-3945-4732-bb5e-515d905fa913\") " pod="openshift-monitoring/node-exporter-2ls7m"
Sep 18 19:39:42.083216 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:42.082951    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/25c36b4f-3ee8-48f3-83aa-b816203e1335-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-mxhbj\" (UID: \"25c36b4f-3ee8-48f3-83aa-b816203e1335\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-mxhbj"
Sep 18 19:39:42.083779 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:42.083422    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2ee2e08b-c234-4675-83ba-a76063ea1285-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-n86fq\" (UID: \"2ee2e08b-c234-4675-83ba-a76063ea1285\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-n86fq"
Sep 18 19:39:42.083779 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:42.083459    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2ee2e08b-c234-4675-83ba-a76063ea1285-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-n86fq\" (UID: \"2ee2e08b-c234-4675-83ba-a76063ea1285\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-n86fq"
Sep 18 19:39:42.083779 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:42.083490    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2ee2e08b-c234-4675-83ba-a76063ea1285-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-n86fq\" (UID: \"2ee2e08b-c234-4675-83ba-a76063ea1285\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-n86fq"
Sep 18 19:39:42.083779 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:42.083512    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d5526936-3945-4732-bb5e-515d905fa913-root\") pod \"node-exporter-2ls7m\" (UID: \"d5526936-3945-4732-bb5e-515d905fa913\") " pod="openshift-monitoring/node-exporter-2ls7m"
Sep 18 19:39:42.083779 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:42.083557    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5526936-3945-4732-bb5e-515d905fa913-metrics-client-ca\") pod \"node-exporter-2ls7m\" (UID: \"d5526936-3945-4732-bb5e-515d905fa913\") " pod="openshift-monitoring/node-exporter-2ls7m"
Sep 18 19:39:42.083779 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:42.083563    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5tqh\" (UniqueName: \"kubernetes.io/projected/2ee2e08b-c234-4675-83ba-a76063ea1285-kube-api-access-g5tqh\") pod \"kube-state-metrics-84c885658c-n86fq\" (UID: \"2ee2e08b-c234-4675-83ba-a76063ea1285\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-n86fq"
Sep 18 19:39:42.083779 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:42.083050    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d5526936-3945-4732-bb5e-515d905fa913-sys\") pod \"node-exporter-2ls7m\" (UID: \"d5526936-3945-4732-bb5e-515d905fa913\") " pod="openshift-monitoring/node-exporter-2ls7m"
Sep 18 19:39:42.083779 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:42.083651    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25c36b4f-3ee8-48f3-83aa-b816203e1335-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-mxhbj\" (UID: \"25c36b4f-3ee8-48f3-83aa-b816203e1335\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-mxhbj"
Sep 18 19:39:42.083779 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:39:42.083027    2564 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 18 19:39:42.083779 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:39:42.083756    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25c36b4f-3ee8-48f3-83aa-b816203e1335-openshift-state-metrics-tls podName:25c36b4f-3ee8-48f3-83aa-b816203e1335 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:42.583735245 +0000 UTC m=+99.094091320 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/25c36b4f-3ee8-48f3-83aa-b816203e1335-openshift-state-metrics-tls") pod "openshift-state-metrics-86c5bc79d6-mxhbj" (UID: "25c36b4f-3ee8-48f3-83aa-b816203e1335") : secret "openshift-state-metrics-tls" not found
Sep 18 19:39:42.083779 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:42.083755    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ee2e08b-c234-4675-83ba-a76063ea1285-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-n86fq\" (UID: \"2ee2e08b-c234-4675-83ba-a76063ea1285\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-n86fq"
Sep 18 19:39:42.083779 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:42.083676    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t652n\" (UniqueName: \"kubernetes.io/projected/25c36b4f-3ee8-48f3-83aa-b816203e1335-kube-api-access-t652n\") pod \"openshift-state-metrics-86c5bc79d6-mxhbj\" (UID: \"25c36b4f-3ee8-48f3-83aa-b816203e1335\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-mxhbj"
Sep 18 19:39:42.084421 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:42.083831    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2ee2e08b-c234-4675-83ba-a76063ea1285-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-n86fq\" (UID: \"2ee2e08b-c234-4675-83ba-a76063ea1285\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-n86fq"
Sep 18 19:39:42.084421 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:42.083859    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d5526936-3945-4732-bb5e-515d905fa913-node-exporter-textfile\") pod \"node-exporter-2ls7m\" (UID: \"d5526936-3945-4732-bb5e-515d905fa913\") " pod="openshift-monitoring/node-exporter-2ls7m"
Sep 18 19:39:42.084421 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:42.083887    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d5526936-3945-4732-bb5e-515d905fa913-node-exporter-wtmp\") pod \"node-exporter-2ls7m\" (UID: \"d5526936-3945-4732-bb5e-515d905fa913\") " pod="openshift-monitoring/node-exporter-2ls7m"
Sep 18 19:39:42.084421 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:42.083918    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d5526936-3945-4732-bb5e-515d905fa913-node-exporter-accelerators-collector-config\") pod \"node-exporter-2ls7m\" (UID: \"d5526936-3945-4732-bb5e-515d905fa913\") " pod="openshift-monitoring/node-exporter-2ls7m"
Sep 18 19:39:42.084421 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:42.083964    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d5526936-3945-4732-bb5e-515d905fa913-node-exporter-tls\") pod \"node-exporter-2ls7m\" (UID: \"d5526936-3945-4732-bb5e-515d905fa913\") " pod="openshift-monitoring/node-exporter-2ls7m"
Sep 18 19:39:42.084421 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:42.084036    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78bfv\" (UniqueName: \"kubernetes.io/projected/d5526936-3945-4732-bb5e-515d905fa913-kube-api-access-78bfv\") pod \"node-exporter-2ls7m\" (UID: \"d5526936-3945-4732-bb5e-515d905fa913\") " pod="openshift-monitoring/node-exporter-2ls7m"
Sep 18 19:39:42.084421 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:42.084185    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d5526936-3945-4732-bb5e-515d905fa913-node-exporter-wtmp\") pod \"node-exporter-2ls7m\" (UID: \"d5526936-3945-4732-bb5e-515d905fa913\") " pod="openshift-monitoring/node-exporter-2ls7m"
Sep 18 19:39:42.084421 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:42.084283    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2ee2e08b-c234-4675-83ba-a76063ea1285-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-n86fq\" (UID: \"2ee2e08b-c234-4675-83ba-a76063ea1285\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-n86fq"
Sep 18 19:39:42.084421 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:42.083917    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d5526936-3945-4732-bb5e-515d905fa913-root\") pod \"node-exporter-2ls7m\" (UID: \"d5526936-3945-4732-bb5e-515d905fa913\") " pod="openshift-monitoring/node-exporter-2ls7m"
Sep 18 19:39:42.084866 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:42.084435    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2ee2e08b-c234-4675-83ba-a76063ea1285-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-n86fq\" (UID: \"2ee2e08b-c234-4675-83ba-a76063ea1285\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-n86fq"
Sep 18 19:39:42.085035 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:42.085011    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d5526936-3945-4732-bb5e-515d905fa913-node-exporter-accelerators-collector-config\") pod \"node-exporter-2ls7m\" (UID: \"d5526936-3945-4732-bb5e-515d905fa913\") " pod="openshift-monitoring/node-exporter-2ls7m"
Sep 18 19:39:42.085144 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:42.085125    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d5526936-3945-4732-bb5e-515d905fa913-node-exporter-textfile\") pod \"node-exporter-2ls7m\" (UID: \"d5526936-3945-4732-bb5e-515d905fa913\") " pod="openshift-monitoring/node-exporter-2ls7m"
Sep 18 19:39:42.085690 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:42.085657    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d5526936-3945-4732-bb5e-515d905fa913-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2ls7m\" (UID: \"d5526936-3945-4732-bb5e-515d905fa913\") " pod="openshift-monitoring/node-exporter-2ls7m"
Sep 18 19:39:42.085930 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:42.085911    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/25c36b4f-3ee8-48f3-83aa-b816203e1335-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-mxhbj\" (UID: \"25c36b4f-3ee8-48f3-83aa-b816203e1335\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-mxhbj"
Sep 18 19:39:42.086279 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:42.086257    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2ee2e08b-c234-4675-83ba-a76063ea1285-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-n86fq\" (UID: \"2ee2e08b-c234-4675-83ba-a76063ea1285\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-n86fq"
Sep 18 19:39:42.086605 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:42.086583    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d5526936-3945-4732-bb5e-515d905fa913-node-exporter-tls\") pod \"node-exporter-2ls7m\" (UID: \"d5526936-3945-4732-bb5e-515d905fa913\") " pod="openshift-monitoring/node-exporter-2ls7m"
Sep 18 19:39:42.086647 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:42.086590    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2ee2e08b-c234-4675-83ba-a76063ea1285-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-n86fq\" (UID: \"2ee2e08b-c234-4675-83ba-a76063ea1285\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-n86fq"
Sep 18 19:39:42.091727 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:42.091700    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78bfv\" (UniqueName: \"kubernetes.io/projected/d5526936-3945-4732-bb5e-515d905fa913-kube-api-access-78bfv\") pod \"node-exporter-2ls7m\" (UID: \"d5526936-3945-4732-bb5e-515d905fa913\") " pod="openshift-monitoring/node-exporter-2ls7m"
Sep 18 19:39:42.091871 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:42.091853    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t652n\" (UniqueName: \"kubernetes.io/projected/25c36b4f-3ee8-48f3-83aa-b816203e1335-kube-api-access-t652n\") pod \"openshift-state-metrics-86c5bc79d6-mxhbj\" (UID: \"25c36b4f-3ee8-48f3-83aa-b816203e1335\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-mxhbj"
Sep 18 19:39:42.091922 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:42.091867    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5tqh\" (UniqueName: \"kubernetes.io/projected/2ee2e08b-c234-4675-83ba-a76063ea1285-kube-api-access-g5tqh\") pod \"kube-state-metrics-84c885658c-n86fq\" (UID: \"2ee2e08b-c234-4675-83ba-a76063ea1285\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-n86fq"
Sep 18 19:39:42.099560 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:42.099537    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5f8sq\""
Sep 18 19:39:42.108273 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:42.108248    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5krlr"
Sep 18 19:39:42.129924 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:42.129895    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2ls7m"
Sep 18 19:39:42.144444 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:42.144407    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-n86fq"
Sep 18 19:39:42.144604 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:39:42.144508    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5526936_3945_4732_bb5e_515d905fa913.slice/crio-12036f2a8e7521d7325aa82ceab0a932db5a1a433c864dc876ed7660c820e02d WatchSource:0}: Error finding container 12036f2a8e7521d7325aa82ceab0a932db5a1a433c864dc876ed7660c820e02d: Status 404 returned error can't find the container with id 12036f2a8e7521d7325aa82ceab0a932db5a1a433c864dc876ed7660c820e02d
Sep 18 19:39:42.174004 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:42.173943    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:42.274784 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:42.274746    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:42.387947 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:42.387778    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5krlr"]
Sep 18 19:39:42.390572 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:39:42.390542    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b6ee93c_e19d_44f2_9ba9_ae133edb7686.slice/crio-6c7851e09a281e55949e7062b786f429129020ddbc339b759672a4e495866c02 WatchSource:0}: Error finding container 6c7851e09a281e55949e7062b786f429129020ddbc339b759672a4e495866c02: Status 404 returned error can't find the container with id 6c7851e09a281e55949e7062b786f429129020ddbc339b759672a4e495866c02
Sep 18 19:39:42.427393 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:42.427364    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-n86fq"]
Sep 18 19:39:42.430584 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:39:42.430559    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ee2e08b_c234_4675_83ba_a76063ea1285.slice/crio-4eb2763247fc68f6e902c8c56b27e203ae145b3a378a604c73dd078e476eda5d WatchSource:0}: Error finding container 4eb2763247fc68f6e902c8c56b27e203ae145b3a378a604c73dd078e476eda5d: Status 404 returned error can't find the container with id 4eb2763247fc68f6e902c8c56b27e203ae145b3a378a604c73dd078e476eda5d
Sep 18 19:39:42.569658 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:42.569620    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2ls7m" event={"ID":"d5526936-3945-4732-bb5e-515d905fa913","Type":"ContainerStarted","Data":"12036f2a8e7521d7325aa82ceab0a932db5a1a433c864dc876ed7660c820e02d"}
Sep 18 19:39:42.571205 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:42.571170    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5krlr" event={"ID":"8b6ee93c-e19d-44f2-9ba9-ae133edb7686","Type":"ContainerStarted","Data":"6c7851e09a281e55949e7062b786f429129020ddbc339b759672a4e495866c02"}
Sep 18 19:39:42.572457 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:42.572434    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-n86fq" event={"ID":"2ee2e08b-c234-4675-83ba-a76063ea1285","Type":"ContainerStarted","Data":"4eb2763247fc68f6e902c8c56b27e203ae145b3a378a604c73dd078e476eda5d"}
Sep 18 19:39:42.588040 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:42.587877    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/25c36b4f-3ee8-48f3-83aa-b816203e1335-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-mxhbj\" (UID: \"25c36b4f-3ee8-48f3-83aa-b816203e1335\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-mxhbj"
Sep 18 19:39:42.590528 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:42.590499    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/25c36b4f-3ee8-48f3-83aa-b816203e1335-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-mxhbj\" (UID: \"25c36b4f-3ee8-48f3-83aa-b816203e1335\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-mxhbj"
Sep 18 19:39:42.636638 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:42.636600    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:42.718860 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:42.718782    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-mxhbj"
Sep 18 19:39:42.808729 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:42.808690    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:42.909526 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:42.909486    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:43.029906 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:43.029842    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-mxhbj"]
Sep 18 19:39:43.035199 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:43.035114    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:43.213334 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:39:43.213290    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25c36b4f_3ee8_48f3_83aa_b816203e1335.slice/crio-fd20c7e9995bb46cf7eb827ee623156124084eb879974fb32ff3d1ad42c6b8bd WatchSource:0}: Error finding container fd20c7e9995bb46cf7eb827ee623156124084eb879974fb32ff3d1ad42c6b8bd: Status 404 returned error can't find the container with id fd20c7e9995bb46cf7eb827ee623156124084eb879974fb32ff3d1ad42c6b8bd
Sep 18 19:39:43.310551 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:43.310446    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:43.408024 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:43.407672    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:43.579898 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:43.579784    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2ls7m" event={"ID":"d5526936-3945-4732-bb5e-515d905fa913","Type":"ContainerStarted","Data":"167f8ebed41509918dda83aa1818893a25afd3fb2c56284e33e20cc0e03c385b"}
Sep 18 19:39:43.582356 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:43.582323    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-mxhbj" event={"ID":"25c36b4f-3ee8-48f3-83aa-b816203e1335","Type":"ContainerStarted","Data":"3d4de4f8ee9f591e5d2d84fcd302fc6fafd8e2e379402a58248fad1cf7d931f2"}
Sep 18 19:39:43.582512 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:43.582393    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-mxhbj" event={"ID":"25c36b4f-3ee8-48f3-83aa-b816203e1335","Type":"ContainerStarted","Data":"fcc9914b249b5389b0361b478f37fa22df66a1240512104dae492570990849cd"}
Sep 18 19:39:43.582512 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:43.582423    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-mxhbj" event={"ID":"25c36b4f-3ee8-48f3-83aa-b816203e1335","Type":"ContainerStarted","Data":"fd20c7e9995bb46cf7eb827ee623156124084eb879974fb32ff3d1ad42c6b8bd"}
Sep 18 19:39:44.589745 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:44.589644    2564 generic.go:356] "Generic (PLEG): container finished" podID="d5526936-3945-4732-bb5e-515d905fa913" containerID="167f8ebed41509918dda83aa1818893a25afd3fb2c56284e33e20cc0e03c385b" exitCode=0
Sep 18 19:39:44.590158 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:44.589774    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2ls7m" event={"ID":"d5526936-3945-4732-bb5e-515d905fa913","Type":"ContainerDied","Data":"167f8ebed41509918dda83aa1818893a25afd3fb2c56284e33e20cc0e03c385b"}
Sep 18 19:39:44.594151 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:44.594076    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5krlr" event={"ID":"8b6ee93c-e19d-44f2-9ba9-ae133edb7686","Type":"ContainerStarted","Data":"b88f87fb8a4908e55aa5d47f8921bf39c0486634433c9250ef77fafcd9d4ee1a"}
Sep 18 19:39:44.594151 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:44.594146    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-5krlr"
Sep 18 19:39:44.594361 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:44.594172    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5krlr" event={"ID":"8b6ee93c-e19d-44f2-9ba9-ae133edb7686","Type":"ContainerStarted","Data":"a1ee145cbeb92e0c7d692e8a9d277e3a1844b27a61381edd89c8b40447d8079f"}
Sep 18 19:39:44.596761 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:44.596670    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-n86fq" event={"ID":"2ee2e08b-c234-4675-83ba-a76063ea1285","Type":"ContainerStarted","Data":"73888e107aa15c51012db76aaf8bf1d286bfff6faf9d26ae6bb018af275f0353"}
Sep 18 19:39:44.596761 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:44.596720    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-n86fq" event={"ID":"2ee2e08b-c234-4675-83ba-a76063ea1285","Type":"ContainerStarted","Data":"c2517e7df639f5df6cae09b84a5e9bc61a8b8772f8dfb9a037c431d56637ead2"}
Sep 18 19:39:44.596761 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:44.596737    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-n86fq" event={"ID":"2ee2e08b-c234-4675-83ba-a76063ea1285","Type":"ContainerStarted","Data":"4411e82caead66b925c07a100480b4ba3240cbb83615676df52c9e39998b9397"}
Sep 18 19:39:44.631058 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:44.630965    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-5krlr" podStartSLOduration=65.747895085 podStartE2EDuration="1m7.630946313s" totalImagesPullingTime="1.883051228s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:38:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:44.629748137 +0000 UTC m=+101.140104252" watchObservedRunningTime="2026-09-18 19:39:44.630946313 +0000 UTC m=+101.141302440"
Sep 18 19:39:44.650635 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:44.650577    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-84c885658c-n86fq" podStartSLOduration=1.8130973049999999 podStartE2EDuration="3.650560068s" totalImagesPullingTime="1.837462763s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:39:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:44.650133084 +0000 UTC m=+101.160489188" watchObservedRunningTime="2026-09-18 19:39:44.650560068 +0000 UTC m=+101.160916195"
Sep 18 19:39:44.771302 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:44.771261    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:44.869108 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:44.869022    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:44.969299 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:44.969264    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:45.070171 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:45.070138    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:45.457456 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:45.457424    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-857bd"
Sep 18 19:39:45.603198 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:45.603165    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-mxhbj" event={"ID":"25c36b4f-3ee8-48f3-83aa-b816203e1335","Type":"ContainerStarted","Data":"b5248167ebac1ee9de805ef5f3bd04d514364a7dba5b308dfeefdc6982b8cd8d"}
Sep 18 19:39:45.605446 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:45.605415    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2ls7m" event={"ID":"d5526936-3945-4732-bb5e-515d905fa913","Type":"ContainerStarted","Data":"158ad89f76ad2b5af5150094d03fb54a8504a07e37222689cc646de9a0118a1e"}
Sep 18 19:39:45.605597 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:45.605465    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2ls7m" event={"ID":"d5526936-3945-4732-bb5e-515d905fa913","Type":"ContainerStarted","Data":"3138cbb6b0a49eb9ade3f4df2bb2b1cbc8a35982abeffbeca74772e6d8634115"}
Sep 18 19:39:45.620034 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:45.619958    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-mxhbj" podStartSLOduration=2.9744741599999998 podStartE2EDuration="4.619941508s" totalImagesPullingTime="1.645467348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:39:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:45.619465567 +0000 UTC m=+102.129821659" watchObservedRunningTime="2026-09-18 19:39:45.619941508 +0000 UTC m=+102.130297598"
Sep 18 19:39:45.639754 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:45.639700    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-2ls7m" podStartSLOduration=3.440529718 podStartE2EDuration="4.639683043s" totalImagesPullingTime="1.120065759s" totalInitContainerRuntime="79.087566ms" isStatefulPod=true podCreationTimestamp="2026-09-18 19:39:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:45.638601795 +0000 UTC m=+102.148957887" watchObservedRunningTime="2026-09-18 19:39:45.639683043 +0000 UTC m=+102.150039134"
Sep 18 19:39:46.293136 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:46.293101    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-565bdf6bdc-qbdl7"]
Sep 18 19:39:46.316954 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:46.316914    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-565bdf6bdc-qbdl7"]
Sep 18 19:39:46.317106 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:46.317069    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-565bdf6bdc-qbdl7"
Sep 18 19:39:46.320164 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:46.320137    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 18 19:39:46.320333 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:46.320248    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 18 19:39:46.320333 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:46.320171    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 18 19:39:46.320599 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:46.320580    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-r57ml\""
Sep 18 19:39:46.320921 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:46.320903    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 18 19:39:46.321008 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:46.320904    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-b5spvakbqaff6\""
Sep 18 19:39:46.421861 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:46.421821    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/73718dc5-55c2-4493-af09-39dd4b81dce2-client-ca-bundle\") pod \"metrics-server-565bdf6bdc-qbdl7\" (UID: \"73718dc5-55c2-4493-af09-39dd4b81dce2\") " pod="openshift-monitoring/metrics-server-565bdf6bdc-qbdl7"
Sep 18 19:39:46.421861 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:46.421860    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73718dc5-55c2-4493-af09-39dd4b81dce2-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-565bdf6bdc-qbdl7\" (UID: \"73718dc5-55c2-4493-af09-39dd4b81dce2\") " pod="openshift-monitoring/metrics-server-565bdf6bdc-qbdl7"
Sep 18 19:39:46.422100 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:46.421892    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/73718dc5-55c2-4493-af09-39dd4b81dce2-secret-metrics-server-tls\") pod \"metrics-server-565bdf6bdc-qbdl7\" (UID: \"73718dc5-55c2-4493-af09-39dd4b81dce2\") " pod="openshift-monitoring/metrics-server-565bdf6bdc-qbdl7"
Sep 18 19:39:46.422100 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:46.421971    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/73718dc5-55c2-4493-af09-39dd4b81dce2-secret-metrics-server-client-certs\") pod \"metrics-server-565bdf6bdc-qbdl7\" (UID: \"73718dc5-55c2-4493-af09-39dd4b81dce2\") " pod="openshift-monitoring/metrics-server-565bdf6bdc-qbdl7"
Sep 18 19:39:46.422100 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:46.422027    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/73718dc5-55c2-4493-af09-39dd4b81dce2-audit-log\") pod \"metrics-server-565bdf6bdc-qbdl7\" (UID: \"73718dc5-55c2-4493-af09-39dd4b81dce2\") " pod="openshift-monitoring/metrics-server-565bdf6bdc-qbdl7"
Sep 18 19:39:46.422100 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:46.422082    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/73718dc5-55c2-4493-af09-39dd4b81dce2-metrics-server-audit-profiles\") pod \"metrics-server-565bdf6bdc-qbdl7\" (UID: \"73718dc5-55c2-4493-af09-39dd4b81dce2\") " pod="openshift-monitoring/metrics-server-565bdf6bdc-qbdl7"
Sep 18 19:39:46.422223 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:46.422104    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j26c6\" (UniqueName: \"kubernetes.io/projected/73718dc5-55c2-4493-af09-39dd4b81dce2-kube-api-access-j26c6\") pod \"metrics-server-565bdf6bdc-qbdl7\" (UID: \"73718dc5-55c2-4493-af09-39dd4b81dce2\") " pod="openshift-monitoring/metrics-server-565bdf6bdc-qbdl7"
Sep 18 19:39:46.523352 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:46.523308    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/73718dc5-55c2-4493-af09-39dd4b81dce2-secret-metrics-server-tls\") pod \"metrics-server-565bdf6bdc-qbdl7\" (UID: \"73718dc5-55c2-4493-af09-39dd4b81dce2\") " pod="openshift-monitoring/metrics-server-565bdf6bdc-qbdl7"
Sep 18 19:39:46.523525 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:46.523359    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/73718dc5-55c2-4493-af09-39dd4b81dce2-secret-metrics-server-client-certs\") pod \"metrics-server-565bdf6bdc-qbdl7\" (UID: \"73718dc5-55c2-4493-af09-39dd4b81dce2\") " pod="openshift-monitoring/metrics-server-565bdf6bdc-qbdl7"
Sep 18 19:39:46.523525 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:46.523389    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/73718dc5-55c2-4493-af09-39dd4b81dce2-audit-log\") pod \"metrics-server-565bdf6bdc-qbdl7\" (UID: \"73718dc5-55c2-4493-af09-39dd4b81dce2\") " pod="openshift-monitoring/metrics-server-565bdf6bdc-qbdl7"
Sep 18 19:39:46.523525 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:46.523437    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/73718dc5-55c2-4493-af09-39dd4b81dce2-metrics-server-audit-profiles\") pod \"metrics-server-565bdf6bdc-qbdl7\" (UID: \"73718dc5-55c2-4493-af09-39dd4b81dce2\") " pod="openshift-monitoring/metrics-server-565bdf6bdc-qbdl7"
Sep 18 19:39:46.523525 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:46.523467    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j26c6\" (UniqueName: \"kubernetes.io/projected/73718dc5-55c2-4493-af09-39dd4b81dce2-kube-api-access-j26c6\") pod \"metrics-server-565bdf6bdc-qbdl7\" (UID: \"73718dc5-55c2-4493-af09-39dd4b81dce2\") " pod="openshift-monitoring/metrics-server-565bdf6bdc-qbdl7"
Sep 18 19:39:46.523707 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:46.523535    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/73718dc5-55c2-4493-af09-39dd4b81dce2-client-ca-bundle\") pod \"metrics-server-565bdf6bdc-qbdl7\" (UID: \"73718dc5-55c2-4493-af09-39dd4b81dce2\") " pod="openshift-monitoring/metrics-server-565bdf6bdc-qbdl7"
Sep 18 19:39:46.523707 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:46.523560    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73718dc5-55c2-4493-af09-39dd4b81dce2-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-565bdf6bdc-qbdl7\" (UID: \"73718dc5-55c2-4493-af09-39dd4b81dce2\") " pod="openshift-monitoring/metrics-server-565bdf6bdc-qbdl7"
Sep 18 19:39:46.523961 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:46.523929    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/73718dc5-55c2-4493-af09-39dd4b81dce2-audit-log\") pod \"metrics-server-565bdf6bdc-qbdl7\" (UID: \"73718dc5-55c2-4493-af09-39dd4b81dce2\") " pod="openshift-monitoring/metrics-server-565bdf6bdc-qbdl7"
Sep 18 19:39:46.524396 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:46.524372    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73718dc5-55c2-4493-af09-39dd4b81dce2-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-565bdf6bdc-qbdl7\" (UID: \"73718dc5-55c2-4493-af09-39dd4b81dce2\") " pod="openshift-monitoring/metrics-server-565bdf6bdc-qbdl7"
Sep 18 19:39:46.524779 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:46.524759    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/73718dc5-55c2-4493-af09-39dd4b81dce2-metrics-server-audit-profiles\") pod \"metrics-server-565bdf6bdc-qbdl7\" (UID: \"73718dc5-55c2-4493-af09-39dd4b81dce2\") " pod="openshift-monitoring/metrics-server-565bdf6bdc-qbdl7"
Sep 18 19:39:46.526197 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:46.526171    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/73718dc5-55c2-4493-af09-39dd4b81dce2-secret-metrics-server-tls\") pod \"metrics-server-565bdf6bdc-qbdl7\" (UID: \"73718dc5-55c2-4493-af09-39dd4b81dce2\") " pod="openshift-monitoring/metrics-server-565bdf6bdc-qbdl7"
Sep 18 19:39:46.526303 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:46.526171    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/73718dc5-55c2-4493-af09-39dd4b81dce2-client-ca-bundle\") pod \"metrics-server-565bdf6bdc-qbdl7\" (UID: \"73718dc5-55c2-4493-af09-39dd4b81dce2\") " pod="openshift-monitoring/metrics-server-565bdf6bdc-qbdl7"
Sep 18 19:39:46.526303 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:46.526209    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/73718dc5-55c2-4493-af09-39dd4b81dce2-secret-metrics-server-client-certs\") pod \"metrics-server-565bdf6bdc-qbdl7\" (UID: \"73718dc5-55c2-4493-af09-39dd4b81dce2\") " pod="openshift-monitoring/metrics-server-565bdf6bdc-qbdl7"
Sep 18 19:39:46.530748 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:46.530721    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j26c6\" (UniqueName: \"kubernetes.io/projected/73718dc5-55c2-4493-af09-39dd4b81dce2-kube-api-access-j26c6\") pod \"metrics-server-565bdf6bdc-qbdl7\" (UID: \"73718dc5-55c2-4493-af09-39dd4b81dce2\") " pod="openshift-monitoring/metrics-server-565bdf6bdc-qbdl7"
Sep 18 19:39:46.592474 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:46.592395    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:46.627203 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:46.627170    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-565bdf6bdc-qbdl7"
Sep 18 19:39:46.930955 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:46.930924    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-565bdf6bdc-qbdl7"]
Sep 18 19:39:46.934384 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:39:46.934352    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73718dc5_55c2_4493_af09_39dd4b81dce2.slice/crio-7069589e32685b63b392fcca624431bbbfb10fd47963feeef90a1f61c263f38b WatchSource:0}: Error finding container 7069589e32685b63b392fcca624431bbbfb10fd47963feeef90a1f61c263f38b: Status 404 returned error can't find the container with id 7069589e32685b63b392fcca624431bbbfb10fd47963feeef90a1f61c263f38b
Sep 18 19:39:47.043771 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:47.043740    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:47.616936 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:47.616893    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-565bdf6bdc-qbdl7" event={"ID":"73718dc5-55c2-4493-af09-39dd4b81dce2","Type":"ContainerStarted","Data":"7069589e32685b63b392fcca624431bbbfb10fd47963feeef90a1f61c263f38b"}
Sep 18 19:39:48.163781 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:48.163737    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:48.411259 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:48.411219    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:48.490433 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:48.490323    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:48.509356 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:48.509326    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-67dc7fd598-rjlqj"
Sep 18 19:39:48.644455 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:48.644414    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:48.742492 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:48.742406    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:49.628448 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:49.628413    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-565bdf6bdc-qbdl7" event={"ID":"73718dc5-55c2-4493-af09-39dd4b81dce2","Type":"ContainerStarted","Data":"ce9ac4dff6976b8eb3dd90ef2dc7960271ab1d359b961e215cf8a8afbde6fb0d"}
Sep 18 19:39:49.646259 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:49.646202    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-565bdf6bdc-qbdl7" podStartSLOduration=1.5219131460000002 podStartE2EDuration="3.646185467s" totalImagesPullingTime="2.124272321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:39:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:49.645264151 +0000 UTC m=+106.155620254" watchObservedRunningTime="2026-09-18 19:39:49.646185467 +0000 UTC m=+106.156541557"
Sep 18 19:39:50.994264 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:50.994226    2564 scope.go:117] "RemoveContainer" containerID="7469e6446d62a75aee2a27153ed14231911828be2895ba72f096a975ec76fa9e"
Sep 18 19:39:51.218106 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:51.218059    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5f57555bf7-2b66f" podUID="7672fd63-163a-481c-aaf8-a1447da0fbf0" containerName="registry" containerID="cri-o://230edfb702f4cc6a64daf5260df832a84165ee1a0bf66a608ce743cc760c4998" gracePeriod=30
Sep 18 19:39:51.454525 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:51.454496    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f57555bf7-2b66f"
Sep 18 19:39:51.574619 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:51.574582    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7672fd63-163a-481c-aaf8-a1447da0fbf0-ca-trust-extracted\") pod \"7672fd63-163a-481c-aaf8-a1447da0fbf0\" (UID: \"7672fd63-163a-481c-aaf8-a1447da0fbf0\") "
Sep 18 19:39:51.574619 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:51.574620    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7672fd63-163a-481c-aaf8-a1447da0fbf0-registry-certificates\") pod \"7672fd63-163a-481c-aaf8-a1447da0fbf0\" (UID: \"7672fd63-163a-481c-aaf8-a1447da0fbf0\") "
Sep 18 19:39:51.574855 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:51.574667    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7672fd63-163a-481c-aaf8-a1447da0fbf0-image-registry-private-configuration\") pod \"7672fd63-163a-481c-aaf8-a1447da0fbf0\" (UID: \"7672fd63-163a-481c-aaf8-a1447da0fbf0\") "
Sep 18 19:39:51.574855 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:51.574690    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7672fd63-163a-481c-aaf8-a1447da0fbf0-bound-sa-token\") pod \"7672fd63-163a-481c-aaf8-a1447da0fbf0\" (UID: \"7672fd63-163a-481c-aaf8-a1447da0fbf0\") "
Sep 18 19:39:51.574855 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:51.574704    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7672fd63-163a-481c-aaf8-a1447da0fbf0-registry-tls\") pod \"7672fd63-163a-481c-aaf8-a1447da0fbf0\" (UID: \"7672fd63-163a-481c-aaf8-a1447da0fbf0\") "
Sep 18 19:39:51.574855 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:51.574734    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7672fd63-163a-481c-aaf8-a1447da0fbf0-trusted-ca\") pod \"7672fd63-163a-481c-aaf8-a1447da0fbf0\" (UID: \"7672fd63-163a-481c-aaf8-a1447da0fbf0\") "
Sep 18 19:39:51.574855 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:51.574758    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5qg4x\" (UniqueName: \"kubernetes.io/projected/7672fd63-163a-481c-aaf8-a1447da0fbf0-kube-api-access-5qg4x\") pod \"7672fd63-163a-481c-aaf8-a1447da0fbf0\" (UID: \"7672fd63-163a-481c-aaf8-a1447da0fbf0\") "
Sep 18 19:39:51.574855 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:51.574808    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7672fd63-163a-481c-aaf8-a1447da0fbf0-installation-pull-secrets\") pod \"7672fd63-163a-481c-aaf8-a1447da0fbf0\" (UID: \"7672fd63-163a-481c-aaf8-a1447da0fbf0\") "
Sep 18 19:39:51.575844 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:51.575242    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7672fd63-163a-481c-aaf8-a1447da0fbf0-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "7672fd63-163a-481c-aaf8-a1447da0fbf0" (UID: "7672fd63-163a-481c-aaf8-a1447da0fbf0"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:39:51.575844 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:51.575613    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7672fd63-163a-481c-aaf8-a1447da0fbf0-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "7672fd63-163a-481c-aaf8-a1447da0fbf0" (UID: "7672fd63-163a-481c-aaf8-a1447da0fbf0"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:39:51.577654 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:51.577624    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7672fd63-163a-481c-aaf8-a1447da0fbf0-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "7672fd63-163a-481c-aaf8-a1447da0fbf0" (UID: "7672fd63-163a-481c-aaf8-a1447da0fbf0"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:39:51.577766 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:51.577718    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7672fd63-163a-481c-aaf8-a1447da0fbf0-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "7672fd63-163a-481c-aaf8-a1447da0fbf0" (UID: "7672fd63-163a-481c-aaf8-a1447da0fbf0"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:39:51.577828 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:51.577762    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7672fd63-163a-481c-aaf8-a1447da0fbf0-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "7672fd63-163a-481c-aaf8-a1447da0fbf0" (UID: "7672fd63-163a-481c-aaf8-a1447da0fbf0"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:39:51.577828 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:51.577812    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7672fd63-163a-481c-aaf8-a1447da0fbf0-kube-api-access-5qg4x" (OuterVolumeSpecName: "kube-api-access-5qg4x") pod "7672fd63-163a-481c-aaf8-a1447da0fbf0" (UID: "7672fd63-163a-481c-aaf8-a1447da0fbf0"). InnerVolumeSpecName "kube-api-access-5qg4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:39:51.577930 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:51.577841    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7672fd63-163a-481c-aaf8-a1447da0fbf0-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "7672fd63-163a-481c-aaf8-a1447da0fbf0" (UID: "7672fd63-163a-481c-aaf8-a1447da0fbf0"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:39:51.583886 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:51.583863    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7672fd63-163a-481c-aaf8-a1447da0fbf0-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "7672fd63-163a-481c-aaf8-a1447da0fbf0" (UID: "7672fd63-163a-481c-aaf8-a1447da0fbf0"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:39:51.639716 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:51.639686    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-58vl4_945a444e-9381-4e2f-b4c1-64be0ec88e6c/console-operator/2.log"
Sep 18 19:39:51.639891 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:51.639791    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-58vl4" event={"ID":"945a444e-9381-4e2f-b4c1-64be0ec88e6c","Type":"ContainerStarted","Data":"bf8362a9ef0cf34cf1e605c2b043dc4d46b361d78818af8f35f0550474ee1bed"}
Sep 18 19:39:51.640161 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:51.640085    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-58vl4"
Sep 18 19:39:51.641272 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:51.641247    2564 generic.go:356] "Generic (PLEG): container finished" podID="7672fd63-163a-481c-aaf8-a1447da0fbf0" containerID="230edfb702f4cc6a64daf5260df832a84165ee1a0bf66a608ce743cc760c4998" exitCode=0
Sep 18 19:39:51.641373 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:51.641289    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f57555bf7-2b66f" event={"ID":"7672fd63-163a-481c-aaf8-a1447da0fbf0","Type":"ContainerDied","Data":"230edfb702f4cc6a64daf5260df832a84165ee1a0bf66a608ce743cc760c4998"}
Sep 18 19:39:51.641373 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:51.641292    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f57555bf7-2b66f"
Sep 18 19:39:51.641373 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:51.641318    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f57555bf7-2b66f" event={"ID":"7672fd63-163a-481c-aaf8-a1447da0fbf0","Type":"ContainerDied","Data":"83b1412ebb63e831976b61fadf3391f1ff86460af0eb275ac54e0314062f0ec3"}
Sep 18 19:39:51.641373 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:51.641343    2564 scope.go:117] "RemoveContainer" containerID="230edfb702f4cc6a64daf5260df832a84165ee1a0bf66a608ce743cc760c4998"
Sep 18 19:39:51.650496 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:51.650476    2564 scope.go:117] "RemoveContainer" containerID="230edfb702f4cc6a64daf5260df832a84165ee1a0bf66a608ce743cc760c4998"
Sep 18 19:39:51.650813 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:39:51.650769    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"230edfb702f4cc6a64daf5260df832a84165ee1a0bf66a608ce743cc760c4998\": container with ID starting with 230edfb702f4cc6a64daf5260df832a84165ee1a0bf66a608ce743cc760c4998 not found: ID does not exist" containerID="230edfb702f4cc6a64daf5260df832a84165ee1a0bf66a608ce743cc760c4998"
Sep 18 19:39:51.650859 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:51.650821    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"230edfb702f4cc6a64daf5260df832a84165ee1a0bf66a608ce743cc760c4998"} err="failed to get container status \"230edfb702f4cc6a64daf5260df832a84165ee1a0bf66a608ce743cc760c4998\": rpc error: code = NotFound desc = could not find container \"230edfb702f4cc6a64daf5260df832a84165ee1a0bf66a608ce743cc760c4998\": container with ID starting with 230edfb702f4cc6a64daf5260df832a84165ee1a0bf66a608ce743cc760c4998 not found: ID does not exist"
Sep 18 19:39:51.658361 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:51.658317    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-fd4bb46c-58vl4" podStartSLOduration=53.342308641 podStartE2EDuration="55.65829862s" totalImagesPullingTime="2.315989979s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:38:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:51.65707024 +0000 UTC m=+108.167426360" watchObservedRunningTime="2026-09-18 19:39:51.65829862 +0000 UTC m=+108.168654711"
Sep 18 19:39:51.669234 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:51.669204    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5f57555bf7-2b66f"]
Sep 18 19:39:51.673393 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:51.673367    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5f57555bf7-2b66f"]
Sep 18 19:39:51.676301 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:51.676274    2564 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7672fd63-163a-481c-aaf8-a1447da0fbf0-registry-certificates\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 19:39:51.676401 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:51.676306    2564 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7672fd63-163a-481c-aaf8-a1447da0fbf0-image-registry-private-configuration\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 19:39:51.676401 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:51.676322    2564 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7672fd63-163a-481c-aaf8-a1447da0fbf0-bound-sa-token\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 19:39:51.676401 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:51.676335    2564 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7672fd63-163a-481c-aaf8-a1447da0fbf0-registry-tls\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 19:39:51.676401 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:51.676347    2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7672fd63-163a-481c-aaf8-a1447da0fbf0-trusted-ca\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 19:39:51.676401 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:51.676359    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qg4x\" (UniqueName: \"kubernetes.io/projected/7672fd63-163a-481c-aaf8-a1447da0fbf0-kube-api-access-5qg4x\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 19:39:51.676401 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:51.676373    2564 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7672fd63-163a-481c-aaf8-a1447da0fbf0-installation-pull-secrets\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 19:39:51.676401 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:51.676385    2564 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7672fd63-163a-481c-aaf8-a1447da0fbf0-ca-trust-extracted\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 19:39:51.845042 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:51.844942    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-fd4bb46c-58vl4"
Sep 18 19:39:51.999594 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:51.999558    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7672fd63-163a-481c-aaf8-a1447da0fbf0" path="/var/lib/kubelet/pods/7672fd63-163a-481c-aaf8-a1447da0fbf0/volumes"
Sep 18 19:39:52.041333 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:52.041299    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6594f4c75-x728t"]
Sep 18 19:39:52.041609 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:52.041597    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7672fd63-163a-481c-aaf8-a1447da0fbf0" containerName="registry"
Sep 18 19:39:52.041686 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:52.041611    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7672fd63-163a-481c-aaf8-a1447da0fbf0" containerName="registry"
Sep 18 19:39:52.041686 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:52.041671    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7672fd63-163a-481c-aaf8-a1447da0fbf0" containerName="registry"
Sep 18 19:39:52.044755 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:52.044730    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-x728t"
Sep 18 19:39:52.045128 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:52.045103    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:52.047198 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:52.047179    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 18 19:39:52.047198 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:52.047193    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 18 19:39:52.047345 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:52.047200    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-s4hlp\""
Sep 18 19:39:52.056809 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:52.056783    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-x728t"]
Sep 18 19:39:52.079509 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:52.079464    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbgj5\" (UniqueName: \"kubernetes.io/projected/946a3d52-df38-4631-b5ce-a94283b4c9a4-kube-api-access-rbgj5\") pod \"downloads-6594f4c75-x728t\" (UID: \"946a3d52-df38-4631-b5ce-a94283b4c9a4\") " pod="openshift-console/downloads-6594f4c75-x728t"
Sep 18 19:39:52.147085 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:52.147049    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:52.180239 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:52.180211    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbgj5\" (UniqueName: \"kubernetes.io/projected/946a3d52-df38-4631-b5ce-a94283b4c9a4-kube-api-access-rbgj5\") pod \"downloads-6594f4c75-x728t\" (UID: \"946a3d52-df38-4631-b5ce-a94283b4c9a4\") " pod="openshift-console/downloads-6594f4c75-x728t"
Sep 18 19:39:52.188796 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:52.188768    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbgj5\" (UniqueName: \"kubernetes.io/projected/946a3d52-df38-4631-b5ce-a94283b4c9a4-kube-api-access-rbgj5\") pod \"downloads-6594f4c75-x728t\" (UID: \"946a3d52-df38-4631-b5ce-a94283b4c9a4\") " pod="openshift-console/downloads-6594f4c75-x728t"
Sep 18 19:39:52.245165 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:52.245130    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:52.346207 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:52.346172    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:52.357125 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:52.357099    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-x728t"
Sep 18 19:39:52.553387 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:52.553349    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:52.633330 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:52.633289    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-x728t"]
Sep 18 19:39:52.637364 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:39:52.637337    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod946a3d52_df38_4631_b5ce_a94283b4c9a4.slice/crio-c8cc8b3c3e2a5a23da52276e3f700c0114876514e9c74b9a0d4b263a2454d977 WatchSource:0}: Error finding container c8cc8b3c3e2a5a23da52276e3f700c0114876514e9c74b9a0d4b263a2454d977: Status 404 returned error can't find the container with id c8cc8b3c3e2a5a23da52276e3f700c0114876514e9c74b9a0d4b263a2454d977
Sep 18 19:39:52.646857 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:52.646828    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-x728t" event={"ID":"946a3d52-df38-4631-b5ce-a94283b4c9a4","Type":"ContainerStarted","Data":"c8cc8b3c3e2a5a23da52276e3f700c0114876514e9c74b9a0d4b263a2454d977"}
Sep 18 19:39:52.700891 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:52.700790    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:52.803779 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:52.803751    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:53.101886 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:53.101795    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:53.187937 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:53.187903    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:53.305792 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:53.305748    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:53.920380 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:53.920339    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:54.118173 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:54.118132    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:54.218898 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:54.218811    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:54.608646 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:54.608559    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5krlr"
Sep 18 19:39:55.402850 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:55.402811    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:59.066635 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:59.066587    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:59.167512 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:39:59.167471    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:00.116765 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:00.116724    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:00.180509 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:00.180478    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:01.937961 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:01.937921    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:03.558679 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:03.558648    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5fff8954fc-x5dg9"]
Sep 18 19:40:03.561557 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:03.561531    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fff8954fc-x5dg9"
Sep 18 19:40:03.564264 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:03.564241    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 18 19:40:03.564701 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:03.564680    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 18 19:40:03.564872 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:03.564852    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 18 19:40:03.565016 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:03.564884    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-hrjgw\""
Sep 18 19:40:03.565016 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:03.564744    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 18 19:40:03.565016 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:03.564760    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 18 19:40:03.571703 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:03.571666    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5fff8954fc-x5dg9"]
Sep 18 19:40:03.694242 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:03.694206    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8c202db5-2e8f-4a35-95e4-1992cfdd95de-console-oauth-config\") pod \"console-5fff8954fc-x5dg9\" (UID: \"8c202db5-2e8f-4a35-95e4-1992cfdd95de\") " pod="openshift-console/console-5fff8954fc-x5dg9"
Sep 18 19:40:03.694422 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:03.694261    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8c202db5-2e8f-4a35-95e4-1992cfdd95de-service-ca\") pod \"console-5fff8954fc-x5dg9\" (UID: \"8c202db5-2e8f-4a35-95e4-1992cfdd95de\") " pod="openshift-console/console-5fff8954fc-x5dg9"
Sep 18 19:40:03.694422 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:03.694349    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm7bq\" (UniqueName: \"kubernetes.io/projected/8c202db5-2e8f-4a35-95e4-1992cfdd95de-kube-api-access-qm7bq\") pod \"console-5fff8954fc-x5dg9\" (UID: \"8c202db5-2e8f-4a35-95e4-1992cfdd95de\") " pod="openshift-console/console-5fff8954fc-x5dg9"
Sep 18 19:40:03.694537 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:03.694469    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8c202db5-2e8f-4a35-95e4-1992cfdd95de-oauth-serving-cert\") pod \"console-5fff8954fc-x5dg9\" (UID: \"8c202db5-2e8f-4a35-95e4-1992cfdd95de\") " pod="openshift-console/console-5fff8954fc-x5dg9"
Sep 18 19:40:03.694537 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:03.694501    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8c202db5-2e8f-4a35-95e4-1992cfdd95de-console-config\") pod \"console-5fff8954fc-x5dg9\" (UID: \"8c202db5-2e8f-4a35-95e4-1992cfdd95de\") " pod="openshift-console/console-5fff8954fc-x5dg9"
Sep 18 19:40:03.694537 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:03.694528    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8c202db5-2e8f-4a35-95e4-1992cfdd95de-console-serving-cert\") pod \"console-5fff8954fc-x5dg9\" (UID: \"8c202db5-2e8f-4a35-95e4-1992cfdd95de\") " pod="openshift-console/console-5fff8954fc-x5dg9"
Sep 18 19:40:03.795951 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:03.795911    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8c202db5-2e8f-4a35-95e4-1992cfdd95de-console-oauth-config\") pod \"console-5fff8954fc-x5dg9\" (UID: \"8c202db5-2e8f-4a35-95e4-1992cfdd95de\") " pod="openshift-console/console-5fff8954fc-x5dg9"
Sep 18 19:40:03.796158 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:03.795959    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8c202db5-2e8f-4a35-95e4-1992cfdd95de-service-ca\") pod \"console-5fff8954fc-x5dg9\" (UID: \"8c202db5-2e8f-4a35-95e4-1992cfdd95de\") " pod="openshift-console/console-5fff8954fc-x5dg9"
Sep 18 19:40:03.796158 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:03.796035    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm7bq\" (UniqueName: \"kubernetes.io/projected/8c202db5-2e8f-4a35-95e4-1992cfdd95de-kube-api-access-qm7bq\") pod \"console-5fff8954fc-x5dg9\" (UID: \"8c202db5-2e8f-4a35-95e4-1992cfdd95de\") " pod="openshift-console/console-5fff8954fc-x5dg9"
Sep 18 19:40:03.796158 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:03.796117    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8c202db5-2e8f-4a35-95e4-1992cfdd95de-oauth-serving-cert\") pod \"console-5fff8954fc-x5dg9\" (UID: \"8c202db5-2e8f-4a35-95e4-1992cfdd95de\") " pod="openshift-console/console-5fff8954fc-x5dg9"
Sep 18 19:40:03.796158 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:03.796150    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8c202db5-2e8f-4a35-95e4-1992cfdd95de-console-config\") pod \"console-5fff8954fc-x5dg9\" (UID: \"8c202db5-2e8f-4a35-95e4-1992cfdd95de\") " pod="openshift-console/console-5fff8954fc-x5dg9"
Sep 18 19:40:03.796360 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:03.796180    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8c202db5-2e8f-4a35-95e4-1992cfdd95de-console-serving-cert\") pod \"console-5fff8954fc-x5dg9\" (UID: \"8c202db5-2e8f-4a35-95e4-1992cfdd95de\") " pod="openshift-console/console-5fff8954fc-x5dg9"
Sep 18 19:40:03.796865 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:03.796815    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8c202db5-2e8f-4a35-95e4-1992cfdd95de-service-ca\") pod \"console-5fff8954fc-x5dg9\" (UID: \"8c202db5-2e8f-4a35-95e4-1992cfdd95de\") " pod="openshift-console/console-5fff8954fc-x5dg9"
Sep 18 19:40:03.796996 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:03.796882    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8c202db5-2e8f-4a35-95e4-1992cfdd95de-oauth-serving-cert\") pod \"console-5fff8954fc-x5dg9\" (UID: \"8c202db5-2e8f-4a35-95e4-1992cfdd95de\") " pod="openshift-console/console-5fff8954fc-x5dg9"
Sep 18 19:40:03.797071 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:03.797045    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8c202db5-2e8f-4a35-95e4-1992cfdd95de-console-config\") pod \"console-5fff8954fc-x5dg9\" (UID: \"8c202db5-2e8f-4a35-95e4-1992cfdd95de\") " pod="openshift-console/console-5fff8954fc-x5dg9"
Sep 18 19:40:03.798751 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:03.798726    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8c202db5-2e8f-4a35-95e4-1992cfdd95de-console-oauth-config\") pod \"console-5fff8954fc-x5dg9\" (UID: \"8c202db5-2e8f-4a35-95e4-1992cfdd95de\") " pod="openshift-console/console-5fff8954fc-x5dg9"
Sep 18 19:40:03.798887 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:03.798867    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8c202db5-2e8f-4a35-95e4-1992cfdd95de-console-serving-cert\") pod \"console-5fff8954fc-x5dg9\" (UID: \"8c202db5-2e8f-4a35-95e4-1992cfdd95de\") " pod="openshift-console/console-5fff8954fc-x5dg9"
Sep 18 19:40:03.804257 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:03.804228    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm7bq\" (UniqueName: \"kubernetes.io/projected/8c202db5-2e8f-4a35-95e4-1992cfdd95de-kube-api-access-qm7bq\") pod \"console-5fff8954fc-x5dg9\" (UID: \"8c202db5-2e8f-4a35-95e4-1992cfdd95de\") " pod="openshift-console/console-5fff8954fc-x5dg9"
Sep 18 19:40:03.877240 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:03.877211    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-hrjgw\""
Sep 18 19:40:03.885930 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:03.885896    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fff8954fc-x5dg9"
Sep 18 19:40:04.530942 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:04.530905    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:04.592319 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:04.592279    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:06.627515 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:06.627470    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-565bdf6bdc-qbdl7"
Sep 18 19:40:06.627969 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:06.627556    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-565bdf6bdc-qbdl7"
Sep 18 19:40:08.072616 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:08.072577    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:08.135949 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:08.135874    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:08.735538 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:08.735510    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:09.016277 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:09.016200    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5fff8954fc-x5dg9"]
Sep 18 19:40:09.019717 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:40:09.019686    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c202db5_2e8f_4a35_95e4_1992cfdd95de.slice/crio-7b19071a061ac1aed1964afba933bcaf36140283624f54ba2efe652efafa3491 WatchSource:0}: Error finding container 7b19071a061ac1aed1964afba933bcaf36140283624f54ba2efe652efafa3491: Status 404 returned error can't find the container with id 7b19071a061ac1aed1964afba933bcaf36140283624f54ba2efe652efafa3491
Sep 18 19:40:09.730166 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:09.730066    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fff8954fc-x5dg9" event={"ID":"8c202db5-2e8f-4a35-95e4-1992cfdd95de","Type":"ContainerStarted","Data":"7b19071a061ac1aed1964afba933bcaf36140283624f54ba2efe652efafa3491"}
Sep 18 19:40:09.737532 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:09.734376    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-x728t" event={"ID":"946a3d52-df38-4631-b5ce-a94283b4c9a4","Type":"ContainerStarted","Data":"c88024f829146255a4deb4fc5751789a4e40cd49c1e798a721b79edfdf038719"}
Sep 18 19:40:09.738822 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:09.737911    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6594f4c75-x728t"
Sep 18 19:40:09.745761 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:09.745701    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6594f4c75-x728t"
Sep 18 19:40:09.752319 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:09.752161    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-6594f4c75-x728t" podStartSLOduration=1.55020472 podStartE2EDuration="17.752140246s" totalImagesPullingTime="16.201935526s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:39:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:40:09.749565158 +0000 UTC m=+126.259921286" watchObservedRunningTime="2026-09-18 19:40:09.752140246 +0000 UTC m=+126.262496337"
Sep 18 19:40:10.178777 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:10.178700    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:10.235412 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:10.235371    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:12.782410 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:12.782360    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/256a72d8-e079-42fb-9426-c53adc283d1f-metrics-certs\") pod \"network-metrics-daemon-xgxbn\" (UID: \"256a72d8-e079-42fb-9426-c53adc283d1f\") " pod="openshift-multus/network-metrics-daemon-xgxbn"
Sep 18 19:40:12.785434 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:12.785382    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/256a72d8-e079-42fb-9426-c53adc283d1f-metrics-certs\") pod \"network-metrics-daemon-xgxbn\" (UID: \"256a72d8-e079-42fb-9426-c53adc283d1f\") " pod="openshift-multus/network-metrics-daemon-xgxbn"
Sep 18 19:40:12.814943 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:12.814707    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8c9m8\""
Sep 18 19:40:12.822726 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:12.822688    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xgxbn"
Sep 18 19:40:12.936022 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:12.935868    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:12.996403 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:12.996313    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:14.686857 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:14.686823    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:14.768560 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:14.768475    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xgxbn"]
Sep 18 19:40:14.771367 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:14.771336    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:14.897993 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:40:14.897951    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod256a72d8_e079_42fb_9426_c53adc283d1f.slice/crio-f4389e38c6c81871056d5035c6f7d2a1dda48a369e05b3e5a81fc0d8bf702773 WatchSource:0}: Error finding container f4389e38c6c81871056d5035c6f7d2a1dda48a369e05b3e5a81fc0d8bf702773: Status 404 returned error can't find the container with id f4389e38c6c81871056d5035c6f7d2a1dda48a369e05b3e5a81fc0d8bf702773
Sep 18 19:40:14.935050 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:14.935010    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:15.036161 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:15.036129    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:15.766811 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:15.766772    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fff8954fc-x5dg9" event={"ID":"8c202db5-2e8f-4a35-95e4-1992cfdd95de","Type":"ContainerStarted","Data":"bc0aed747fa7ee7649a6ca2baea3b5c294a2a7d13f4e728bdd567d49fc4c2033"}
Sep 18 19:40:15.768733 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:15.768694    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xgxbn" event={"ID":"256a72d8-e079-42fb-9426-c53adc283d1f","Type":"ContainerStarted","Data":"f4389e38c6c81871056d5035c6f7d2a1dda48a369e05b3e5a81fc0d8bf702773"}
Sep 18 19:40:15.784403 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:15.784332    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5fff8954fc-x5dg9" podStartSLOduration=6.865858945 podStartE2EDuration="12.784308625s" totalImagesPullingTime="5.91844968s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:40:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:40:15.782966101 +0000 UTC m=+132.293322215" watchObservedRunningTime="2026-09-18 19:40:15.784308625 +0000 UTC m=+132.294664718"
Sep 18 19:40:16.776023 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:16.775955    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xgxbn" event={"ID":"256a72d8-e079-42fb-9426-c53adc283d1f","Type":"ContainerStarted","Data":"6b3cc903be77f687f3fd02c3ffdb47aa6a6de3510c65613b5d0d206f8bcc4a7b"}
Sep 18 19:40:17.782850 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:17.782810    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xgxbn" event={"ID":"256a72d8-e079-42fb-9426-c53adc283d1f","Type":"ContainerStarted","Data":"0b8b1578c8662de25ea8ba4ee8b5b7a9db4058233ff791665cd8224c531f6b29"}
Sep 18 19:40:17.797242 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:17.797172    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xgxbn" podStartSLOduration=132.235630404 podStartE2EDuration="2m13.797147503s" totalImagesPullingTime="1.561517099s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:38:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:40:17.796560949 +0000 UTC m=+134.306917093" watchObservedRunningTime="2026-09-18 19:40:17.797147503 +0000 UTC m=+134.307503606"
Sep 18 19:40:17.891136 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:17.891094    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:17.992301 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:17.992263    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:23.886559 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:23.886526    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5fff8954fc-x5dg9"
Sep 18 19:40:23.887029 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:23.886588    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5fff8954fc-x5dg9"
Sep 18 19:40:23.891803 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:23.891777    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5fff8954fc-x5dg9"
Sep 18 19:40:24.199963 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:24.199881    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5fff8954fc-x5dg9"]
Sep 18 19:40:24.541102 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:24.540957    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:24.596812 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:24.596771    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:24.813714 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:24.813623    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5fff8954fc-x5dg9"
Sep 18 19:40:25.141942 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:25.141901    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:25.207935 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:25.207898    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:25.749045 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:25.749008    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:25.835944 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:25.835900    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:26.598481 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:26.598443    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:26.635267 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:26.635228    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-565bdf6bdc-qbdl7"
Sep 18 19:40:26.643792 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:26.643761    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-565bdf6bdc-qbdl7"
Sep 18 19:40:26.655586 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:26.655556    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:29.690383 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:29.690343    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:29.745664 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:29.745633    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:30.845273 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:30.845236    2564 generic.go:356] "Generic (PLEG): container finished" podID="9b05425f-3a0e-40b6-b1af-4f3098050245" containerID="9651a802fec1232c271830bfdca7ebe0e25c80b29d1053b1f826db77406e3a24" exitCode=0
Sep 18 19:40:30.845698 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:30.845314    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-lshmd" event={"ID":"9b05425f-3a0e-40b6-b1af-4f3098050245","Type":"ContainerDied","Data":"9651a802fec1232c271830bfdca7ebe0e25c80b29d1053b1f826db77406e3a24"}
Sep 18 19:40:30.845768 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:30.845751    2564 scope.go:117] "RemoveContainer" containerID="9651a802fec1232c271830bfdca7ebe0e25c80b29d1053b1f826db77406e3a24"
Sep 18 19:40:31.851334 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:31.851302    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-lshmd" event={"ID":"9b05425f-3a0e-40b6-b1af-4f3098050245","Type":"ContainerStarted","Data":"b8b8bc93b42af13816a3a0582776fa941a5a8723c55c56fb0dcba5bbac0f4836"}
Sep 18 19:40:44.691180 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:44.691140    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:44.791742 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:44.791704    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:50.836841 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:50.836792    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5fff8954fc-x5dg9" podUID="8c202db5-2e8f-4a35-95e4-1992cfdd95de" containerName="console" containerID="cri-o://bc0aed747fa7ee7649a6ca2baea3b5c294a2a7d13f4e728bdd567d49fc4c2033" gracePeriod=15
Sep 18 19:40:51.104056 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:51.103945    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fff8954fc-x5dg9"
Sep 18 19:40:51.220187 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:51.220151    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8c202db5-2e8f-4a35-95e4-1992cfdd95de-console-config\") pod \"8c202db5-2e8f-4a35-95e4-1992cfdd95de\" (UID: \"8c202db5-2e8f-4a35-95e4-1992cfdd95de\") "
Sep 18 19:40:51.220385 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:51.220201    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8c202db5-2e8f-4a35-95e4-1992cfdd95de-console-oauth-config\") pod \"8c202db5-2e8f-4a35-95e4-1992cfdd95de\" (UID: \"8c202db5-2e8f-4a35-95e4-1992cfdd95de\") "
Sep 18 19:40:51.220385 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:51.220225    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qm7bq\" (UniqueName: \"kubernetes.io/projected/8c202db5-2e8f-4a35-95e4-1992cfdd95de-kube-api-access-qm7bq\") pod \"8c202db5-2e8f-4a35-95e4-1992cfdd95de\" (UID: \"8c202db5-2e8f-4a35-95e4-1992cfdd95de\") "
Sep 18 19:40:51.220385 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:51.220241    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8c202db5-2e8f-4a35-95e4-1992cfdd95de-console-serving-cert\") pod \"8c202db5-2e8f-4a35-95e4-1992cfdd95de\" (UID: \"8c202db5-2e8f-4a35-95e4-1992cfdd95de\") "
Sep 18 19:40:51.220385 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:51.220264    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8c202db5-2e8f-4a35-95e4-1992cfdd95de-oauth-serving-cert\") pod \"8c202db5-2e8f-4a35-95e4-1992cfdd95de\" (UID: \"8c202db5-2e8f-4a35-95e4-1992cfdd95de\") "
Sep 18 19:40:51.220385 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:51.220303    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8c202db5-2e8f-4a35-95e4-1992cfdd95de-service-ca\") pod \"8c202db5-2e8f-4a35-95e4-1992cfdd95de\" (UID: \"8c202db5-2e8f-4a35-95e4-1992cfdd95de\") "
Sep 18 19:40:51.220744 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:51.220709    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c202db5-2e8f-4a35-95e4-1992cfdd95de-console-config" (OuterVolumeSpecName: "console-config") pod "8c202db5-2e8f-4a35-95e4-1992cfdd95de" (UID: "8c202db5-2e8f-4a35-95e4-1992cfdd95de"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:40:51.220875 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:51.220711    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c202db5-2e8f-4a35-95e4-1992cfdd95de-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "8c202db5-2e8f-4a35-95e4-1992cfdd95de" (UID: "8c202db5-2e8f-4a35-95e4-1992cfdd95de"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:40:51.220875 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:51.220781    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c202db5-2e8f-4a35-95e4-1992cfdd95de-service-ca" (OuterVolumeSpecName: "service-ca") pod "8c202db5-2e8f-4a35-95e4-1992cfdd95de" (UID: "8c202db5-2e8f-4a35-95e4-1992cfdd95de"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:40:51.222639 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:51.222610    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c202db5-2e8f-4a35-95e4-1992cfdd95de-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "8c202db5-2e8f-4a35-95e4-1992cfdd95de" (UID: "8c202db5-2e8f-4a35-95e4-1992cfdd95de"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:40:51.222935 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:51.222913    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c202db5-2e8f-4a35-95e4-1992cfdd95de-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "8c202db5-2e8f-4a35-95e4-1992cfdd95de" (UID: "8c202db5-2e8f-4a35-95e4-1992cfdd95de"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:40:51.223040 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:51.222937    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c202db5-2e8f-4a35-95e4-1992cfdd95de-kube-api-access-qm7bq" (OuterVolumeSpecName: "kube-api-access-qm7bq") pod "8c202db5-2e8f-4a35-95e4-1992cfdd95de" (UID: "8c202db5-2e8f-4a35-95e4-1992cfdd95de"). InnerVolumeSpecName "kube-api-access-qm7bq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:40:51.320804 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:51.320777    2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8c202db5-2e8f-4a35-95e4-1992cfdd95de-service-ca\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 19:40:51.320804 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:51.320800    2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8c202db5-2e8f-4a35-95e4-1992cfdd95de-console-config\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 19:40:51.320804 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:51.320810    2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8c202db5-2e8f-4a35-95e4-1992cfdd95de-console-oauth-config\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 19:40:51.321047 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:51.320820    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qm7bq\" (UniqueName: \"kubernetes.io/projected/8c202db5-2e8f-4a35-95e4-1992cfdd95de-kube-api-access-qm7bq\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 19:40:51.321047 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:51.320829    2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8c202db5-2e8f-4a35-95e4-1992cfdd95de-console-serving-cert\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 19:40:51.321047 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:51.320838    2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8c202db5-2e8f-4a35-95e4-1992cfdd95de-oauth-serving-cert\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 19:40:51.932409 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:51.932372    2564 generic.go:356] "Generic (PLEG): container finished" podID="8c202db5-2e8f-4a35-95e4-1992cfdd95de" containerID="bc0aed747fa7ee7649a6ca2baea3b5c294a2a7d13f4e728bdd567d49fc4c2033" exitCode=0
Sep 18 19:40:51.932814 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:51.932431    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fff8954fc-x5dg9"
Sep 18 19:40:51.932814 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:51.932458    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fff8954fc-x5dg9" event={"ID":"8c202db5-2e8f-4a35-95e4-1992cfdd95de","Type":"ContainerDied","Data":"bc0aed747fa7ee7649a6ca2baea3b5c294a2a7d13f4e728bdd567d49fc4c2033"}
Sep 18 19:40:51.932814 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:51.932534    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fff8954fc-x5dg9" event={"ID":"8c202db5-2e8f-4a35-95e4-1992cfdd95de","Type":"ContainerDied","Data":"7b19071a061ac1aed1964afba933bcaf36140283624f54ba2efe652efafa3491"}
Sep 18 19:40:51.932814 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:51.932555    2564 scope.go:117] "RemoveContainer" containerID="bc0aed747fa7ee7649a6ca2baea3b5c294a2a7d13f4e728bdd567d49fc4c2033"
Sep 18 19:40:51.941445 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:51.941422    2564 scope.go:117] "RemoveContainer" containerID="bc0aed747fa7ee7649a6ca2baea3b5c294a2a7d13f4e728bdd567d49fc4c2033"
Sep 18 19:40:51.941742 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:40:51.941725    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc0aed747fa7ee7649a6ca2baea3b5c294a2a7d13f4e728bdd567d49fc4c2033\": container with ID starting with bc0aed747fa7ee7649a6ca2baea3b5c294a2a7d13f4e728bdd567d49fc4c2033 not found: ID does not exist" containerID="bc0aed747fa7ee7649a6ca2baea3b5c294a2a7d13f4e728bdd567d49fc4c2033"
Sep 18 19:40:51.941790 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:51.941751    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc0aed747fa7ee7649a6ca2baea3b5c294a2a7d13f4e728bdd567d49fc4c2033"} err="failed to get container status \"bc0aed747fa7ee7649a6ca2baea3b5c294a2a7d13f4e728bdd567d49fc4c2033\": rpc error: code = NotFound desc = could not find container \"bc0aed747fa7ee7649a6ca2baea3b5c294a2a7d13f4e728bdd567d49fc4c2033\": container with ID starting with bc0aed747fa7ee7649a6ca2baea3b5c294a2a7d13f4e728bdd567d49fc4c2033 not found: ID does not exist"
Sep 18 19:40:51.952248 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:51.952215    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5fff8954fc-x5dg9"]
Sep 18 19:40:51.954250 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:51.954224    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5fff8954fc-x5dg9"]
Sep 18 19:40:51.999931 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:40:51.999896    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c202db5-2e8f-4a35-95e4-1992cfdd95de" path="/var/lib/kubelet/pods/8c202db5-2e8f-4a35-95e4-1992cfdd95de/volumes"
Sep 18 19:41:01.092254 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:41:01.092155    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:41:01.408365 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:41:01.408327    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:41:09.508865 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:41:09.508831    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:41:09.608829 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:41:09.608791    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:41:20.536309 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:41:20.536263    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:41:20.591477 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:41:20.591433    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:41:42.379190 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:41:42.379101    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:41:42.429498 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:41:42.429447    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:41:43.013303 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:41:43.013263    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:09.644265 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:42:09.644214    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:16.356951 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:42:16.356909    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:16.420093 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:42:16.420062    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:18.662839 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:42:18.662806    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:28.265472 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:42:28.265374    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:28.365888 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:42:28.365845    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:44.694818 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:42:44.694775    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:44.798681 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:42:44.798640    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:52.031110 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:42:52.031072    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7b597c4d4f-jnjz9"]
Sep 18 19:42:52.031645 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:42:52.031559    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c202db5-2e8f-4a35-95e4-1992cfdd95de" containerName="console"
Sep 18 19:42:52.031645 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:42:52.031579    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c202db5-2e8f-4a35-95e4-1992cfdd95de" containerName="console"
Sep 18 19:42:52.031785 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:42:52.031715    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c202db5-2e8f-4a35-95e4-1992cfdd95de" containerName="console"
Sep 18 19:42:52.033826 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:42:52.033803    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b597c4d4f-jnjz9"
Sep 18 19:42:52.035992 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:42:52.035956    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 18 19:42:52.036130 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:42:52.035972    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 18 19:42:52.036668 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:42:52.036637    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 18 19:42:52.036778 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:42:52.036711    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 18 19:42:52.037265 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:42:52.037243    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-hrjgw\""
Sep 18 19:42:52.037471 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:42:52.037456    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 18 19:42:52.041740 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:42:52.041520    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 18 19:42:52.047226 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:42:52.047205    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b597c4d4f-jnjz9"]
Sep 18 19:42:52.117392 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:42:52.117362    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:52.125367 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:42:52.125340    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f36fcc01-5634-4f3c-8c82-3b982107d739-console-oauth-config\") pod \"console-7b597c4d4f-jnjz9\" (UID: \"f36fcc01-5634-4f3c-8c82-3b982107d739\") " pod="openshift-console/console-7b597c4d4f-jnjz9"
Sep 18 19:42:52.125481 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:42:52.125381    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f36fcc01-5634-4f3c-8c82-3b982107d739-console-config\") pod \"console-7b597c4d4f-jnjz9\" (UID: \"f36fcc01-5634-4f3c-8c82-3b982107d739\") " pod="openshift-console/console-7b597c4d4f-jnjz9"
Sep 18 19:42:52.125481 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:42:52.125405    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f36fcc01-5634-4f3c-8c82-3b982107d739-service-ca\") pod \"console-7b597c4d4f-jnjz9\" (UID: \"f36fcc01-5634-4f3c-8c82-3b982107d739\") " pod="openshift-console/console-7b597c4d4f-jnjz9"
Sep 18 19:42:52.125481 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:42:52.125426    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f36fcc01-5634-4f3c-8c82-3b982107d739-oauth-serving-cert\") pod \"console-7b597c4d4f-jnjz9\" (UID: \"f36fcc01-5634-4f3c-8c82-3b982107d739\") " pod="openshift-console/console-7b597c4d4f-jnjz9"
Sep 18 19:42:52.125591 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:42:52.125494    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f36fcc01-5634-4f3c-8c82-3b982107d739-trusted-ca-bundle\") pod \"console-7b597c4d4f-jnjz9\" (UID: \"f36fcc01-5634-4f3c-8c82-3b982107d739\") " pod="openshift-console/console-7b597c4d4f-jnjz9"
Sep 18 19:42:52.125591 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:42:52.125519    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f36fcc01-5634-4f3c-8c82-3b982107d739-console-serving-cert\") pod \"console-7b597c4d4f-jnjz9\" (UID: \"f36fcc01-5634-4f3c-8c82-3b982107d739\") " pod="openshift-console/console-7b597c4d4f-jnjz9"
Sep 18 19:42:52.125591 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:42:52.125533    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bch58\" (UniqueName: \"kubernetes.io/projected/f36fcc01-5634-4f3c-8c82-3b982107d739-kube-api-access-bch58\") pod \"console-7b597c4d4f-jnjz9\" (UID: \"f36fcc01-5634-4f3c-8c82-3b982107d739\") " pod="openshift-console/console-7b597c4d4f-jnjz9"
Sep 18 19:42:52.176323 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:42:52.176288    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:52.226278 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:42:52.226241    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f36fcc01-5634-4f3c-8c82-3b982107d739-trusted-ca-bundle\") pod \"console-7b597c4d4f-jnjz9\" (UID: \"f36fcc01-5634-4f3c-8c82-3b982107d739\") " pod="openshift-console/console-7b597c4d4f-jnjz9"
Sep 18 19:42:52.226278 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:42:52.226282    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f36fcc01-5634-4f3c-8c82-3b982107d739-console-serving-cert\") pod \"console-7b597c4d4f-jnjz9\" (UID: \"f36fcc01-5634-4f3c-8c82-3b982107d739\") " pod="openshift-console/console-7b597c4d4f-jnjz9"
Sep 18 19:42:52.226540 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:42:52.226298    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bch58\" (UniqueName: \"kubernetes.io/projected/f36fcc01-5634-4f3c-8c82-3b982107d739-kube-api-access-bch58\") pod \"console-7b597c4d4f-jnjz9\" (UID: \"f36fcc01-5634-4f3c-8c82-3b982107d739\") " pod="openshift-console/console-7b597c4d4f-jnjz9"
Sep 18 19:42:52.226540 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:42:52.226337    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f36fcc01-5634-4f3c-8c82-3b982107d739-console-oauth-config\") pod \"console-7b597c4d4f-jnjz9\" (UID: \"f36fcc01-5634-4f3c-8c82-3b982107d739\") " pod="openshift-console/console-7b597c4d4f-jnjz9"
Sep 18 19:42:52.226540 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:42:52.226357    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f36fcc01-5634-4f3c-8c82-3b982107d739-console-config\") pod \"console-7b597c4d4f-jnjz9\" (UID: \"f36fcc01-5634-4f3c-8c82-3b982107d739\") " pod="openshift-console/console-7b597c4d4f-jnjz9"
Sep 18 19:42:52.226540 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:42:52.226373    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f36fcc01-5634-4f3c-8c82-3b982107d739-service-ca\") pod \"console-7b597c4d4f-jnjz9\" (UID: \"f36fcc01-5634-4f3c-8c82-3b982107d739\") " pod="openshift-console/console-7b597c4d4f-jnjz9"
Sep 18 19:42:52.226540 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:42:52.226389    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f36fcc01-5634-4f3c-8c82-3b982107d739-oauth-serving-cert\") pod \"console-7b597c4d4f-jnjz9\" (UID: \"f36fcc01-5634-4f3c-8c82-3b982107d739\") " pod="openshift-console/console-7b597c4d4f-jnjz9"
Sep 18 19:42:52.227301 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:42:52.227275    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f36fcc01-5634-4f3c-8c82-3b982107d739-console-config\") pod \"console-7b597c4d4f-jnjz9\" (UID: \"f36fcc01-5634-4f3c-8c82-3b982107d739\") " pod="openshift-console/console-7b597c4d4f-jnjz9"
Sep 18 19:42:52.227406 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:42:52.227282    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f36fcc01-5634-4f3c-8c82-3b982107d739-service-ca\") pod \"console-7b597c4d4f-jnjz9\" (UID: \"f36fcc01-5634-4f3c-8c82-3b982107d739\") " pod="openshift-console/console-7b597c4d4f-jnjz9"
Sep 18 19:42:52.227406 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:42:52.227282    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f36fcc01-5634-4f3c-8c82-3b982107d739-oauth-serving-cert\") pod \"console-7b597c4d4f-jnjz9\" (UID: \"f36fcc01-5634-4f3c-8c82-3b982107d739\") " pod="openshift-console/console-7b597c4d4f-jnjz9"
Sep 18 19:42:52.227489 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:42:52.227418    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f36fcc01-5634-4f3c-8c82-3b982107d739-trusted-ca-bundle\") pod \"console-7b597c4d4f-jnjz9\" (UID: \"f36fcc01-5634-4f3c-8c82-3b982107d739\") " pod="openshift-console/console-7b597c4d4f-jnjz9"
Sep 18 19:42:52.228840 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:42:52.228819    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f36fcc01-5634-4f3c-8c82-3b982107d739-console-serving-cert\") pod \"console-7b597c4d4f-jnjz9\" (UID: \"f36fcc01-5634-4f3c-8c82-3b982107d739\") " pod="openshift-console/console-7b597c4d4f-jnjz9"
Sep 18 19:42:52.228936 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:42:52.228867    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f36fcc01-5634-4f3c-8c82-3b982107d739-console-oauth-config\") pod \"console-7b597c4d4f-jnjz9\" (UID: \"f36fcc01-5634-4f3c-8c82-3b982107d739\") " pod="openshift-console/console-7b597c4d4f-jnjz9"
Sep 18 19:42:52.234394 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:42:52.234370    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bch58\" (UniqueName: \"kubernetes.io/projected/f36fcc01-5634-4f3c-8c82-3b982107d739-kube-api-access-bch58\") pod \"console-7b597c4d4f-jnjz9\" (UID: \"f36fcc01-5634-4f3c-8c82-3b982107d739\") " pod="openshift-console/console-7b597c4d4f-jnjz9"
Sep 18 19:42:52.344652 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:42:52.344539    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b597c4d4f-jnjz9"
Sep 18 19:42:52.618564 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:42:52.618534    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b597c4d4f-jnjz9"]
Sep 18 19:42:52.621211 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:42:52.621183    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf36fcc01_5634_4f3c_8c82_3b982107d739.slice/crio-b4de2e14ad3547664ac6c4df1ba0a12c6238c135f54279dd85fbf3685cbcaa54 WatchSource:0}: Error finding container b4de2e14ad3547664ac6c4df1ba0a12c6238c135f54279dd85fbf3685cbcaa54: Status 404 returned error can't find the container with id b4de2e14ad3547664ac6c4df1ba0a12c6238c135f54279dd85fbf3685cbcaa54
Sep 18 19:42:53.391400 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:42:53.391369    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b597c4d4f-jnjz9" event={"ID":"f36fcc01-5634-4f3c-8c82-3b982107d739","Type":"ContainerStarted","Data":"14de7e9ff7284e6adfb74d4432496fa0a06aeba37068fd98cd0fcb39a2a04588"}
Sep 18 19:42:53.391800 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:42:53.391416    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b597c4d4f-jnjz9" event={"ID":"f36fcc01-5634-4f3c-8c82-3b982107d739","Type":"ContainerStarted","Data":"b4de2e14ad3547664ac6c4df1ba0a12c6238c135f54279dd85fbf3685cbcaa54"}
Sep 18 19:42:53.407894 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:42:53.407833    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7b597c4d4f-jnjz9" podStartSLOduration=1.407814325 podStartE2EDuration="1.407814325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:42:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:42:53.406861508 +0000 UTC m=+289.917217616" watchObservedRunningTime="2026-09-18 19:42:53.407814325 +0000 UTC m=+289.918170415"
Sep 18 19:43:02.345497 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:43:02.345442    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7b597c4d4f-jnjz9"
Sep 18 19:43:02.345497 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:43:02.345516    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7b597c4d4f-jnjz9"
Sep 18 19:43:02.350752 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:43:02.350726    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7b597c4d4f-jnjz9"
Sep 18 19:43:02.427768 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:43:02.427739    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7b597c4d4f-jnjz9"
Sep 18 19:43:02.577048 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:43:02.577008    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:02.678253 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:43:02.678217    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:02.779703 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:43:02.779672    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:03.878878 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:43:03.878850    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-58vl4_945a444e-9381-4e2f-b4c1-64be0ec88e6c/console-operator/2.log"
Sep 18 19:43:03.882783 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:43:03.882755    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-58vl4_945a444e-9381-4e2f-b4c1-64be0ec88e6c/console-operator/2.log"
Sep 18 19:43:03.889890 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:43:03.889867    2564 kubelet.go:1648] "Image garbage collection succeeded"
Sep 18 19:43:27.525992 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:43:27.525940    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:28.895955 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:43:28.895900    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:48.486645 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:43:48.486611    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:49.634916 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:43:49.634885    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:49.725893 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:43:49.725857    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:51.253457 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:43:51.253410    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-7q7km"]
Sep 18 19:43:51.255907 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:43:51.255886    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-7q7km"
Sep 18 19:43:51.257917 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:43:51.257890    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 18 19:43:51.258067 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:43:51.257893    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 18 19:43:51.258067 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:43:51.257895    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-dhxzq\""
Sep 18 19:43:51.258067 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:43:51.258003    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 18 19:43:51.268087 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:43:51.268059    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-7q7km"]
Sep 18 19:43:51.314690 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:43:51.314658    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w5b4\" (UniqueName: \"kubernetes.io/projected/7cd26fda-f3db-45c3-a657-43bf2673af82-kube-api-access-7w5b4\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-7q7km\" (UID: \"7cd26fda-f3db-45c3-a657-43bf2673af82\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-7q7km"
Sep 18 19:43:51.314942 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:43:51.314738    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/7cd26fda-f3db-45c3-a657-43bf2673af82-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-7q7km\" (UID: \"7cd26fda-f3db-45c3-a657-43bf2673af82\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-7q7km"
Sep 18 19:43:51.415189 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:43:51.415148    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/7cd26fda-f3db-45c3-a657-43bf2673af82-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-7q7km\" (UID: \"7cd26fda-f3db-45c3-a657-43bf2673af82\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-7q7km"
Sep 18 19:43:51.415370 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:43:51.415219    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w5b4\" (UniqueName: \"kubernetes.io/projected/7cd26fda-f3db-45c3-a657-43bf2673af82-kube-api-access-7w5b4\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-7q7km\" (UID: \"7cd26fda-f3db-45c3-a657-43bf2673af82\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-7q7km"
Sep 18 19:43:51.417549 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:43:51.417531    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/7cd26fda-f3db-45c3-a657-43bf2673af82-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-7q7km\" (UID: \"7cd26fda-f3db-45c3-a657-43bf2673af82\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-7q7km"
Sep 18 19:43:51.426039 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:43:51.426013    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w5b4\" (UniqueName: \"kubernetes.io/projected/7cd26fda-f3db-45c3-a657-43bf2673af82-kube-api-access-7w5b4\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-7q7km\" (UID: \"7cd26fda-f3db-45c3-a657-43bf2673af82\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-7q7km"
Sep 18 19:43:51.566428 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:43:51.566341    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-7q7km"
Sep 18 19:43:51.841714 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:43:51.841559    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-7q7km"]
Sep 18 19:43:51.844680 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:43:51.844651    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cd26fda_f3db_45c3_a657_43bf2673af82.slice/crio-e9fcc4a84aa7d00282d51c472a4b799dbc3c09896dfb6a7caa444ea47d15d92c WatchSource:0}: Error finding container e9fcc4a84aa7d00282d51c472a4b799dbc3c09896dfb6a7caa444ea47d15d92c: Status 404 returned error can't find the container with id e9fcc4a84aa7d00282d51c472a4b799dbc3c09896dfb6a7caa444ea47d15d92c
Sep 18 19:43:51.846458 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:43:51.846436    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:43:52.628949 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:43:52.628864    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-7q7km" event={"ID":"7cd26fda-f3db-45c3-a657-43bf2673af82","Type":"ContainerStarted","Data":"e9fcc4a84aa7d00282d51c472a4b799dbc3c09896dfb6a7caa444ea47d15d92c"}
Sep 18 19:43:55.262670 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:43:55.262632    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:55.546643 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:43:55.546544    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:55.645710 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:43:55.645673    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-7q7km" event={"ID":"7cd26fda-f3db-45c3-a657-43bf2673af82","Type":"ContainerStarted","Data":"0072a2bc88b9bbe02b93cad53bf0f2e756a5e7b39a6161d765ad383aad141233"}
Sep 18 19:43:55.645950 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:43:55.645847    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-7q7km"
Sep 18 19:43:55.670474 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:43:55.670387    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-7q7km" podStartSLOduration=1.702297251 podStartE2EDuration="4.670368953s" totalImagesPullingTime="2.968071702s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:43:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:43:55.669120344 +0000 UTC m=+352.179476454" watchObservedRunningTime="2026-09-18 19:43:55.670368953 +0000 UTC m=+352.180725044"
Sep 18 19:43:55.694187 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:43:55.694158    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:55.915231 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:43:55.915194    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:56.094935 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:43:56.094858    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-dj7tc"]
Sep 18 19:43:56.097395 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:43:56.097372    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-dj7tc"
Sep 18 19:43:56.099407 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:43:56.099385    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Sep 18 19:43:56.099546 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:43:56.099433    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 18 19:43:56.099660 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:43:56.099648    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-74cfb\""
Sep 18 19:43:56.106560 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:43:56.106532    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-dj7tc"]
Sep 18 19:43:56.143564 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:43:56.143524    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:56.163183 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:43:56.163152    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/80d81101-236a-4598-9ae0-f75d5dcb6e0b-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-dj7tc\" (UID: \"80d81101-236a-4598-9ae0-f75d5dcb6e0b\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-dj7tc"
Sep 18 19:43:56.163367 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:43:56.163195    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/80d81101-236a-4598-9ae0-f75d5dcb6e0b-certificates\") pod \"keda-metrics-apiserver-77857b7b98-dj7tc\" (UID: \"80d81101-236a-4598-9ae0-f75d5dcb6e0b\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-dj7tc"
Sep 18 19:43:56.163367 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:43:56.163279    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmzgq\" (UniqueName: \"kubernetes.io/projected/80d81101-236a-4598-9ae0-f75d5dcb6e0b-kube-api-access-xmzgq\") pod \"keda-metrics-apiserver-77857b7b98-dj7tc\" (UID: \"80d81101-236a-4598-9ae0-f75d5dcb6e0b\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-dj7tc"
Sep 18 19:43:56.264309 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:43:56.264211    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/80d81101-236a-4598-9ae0-f75d5dcb6e0b-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-dj7tc\" (UID: \"80d81101-236a-4598-9ae0-f75d5dcb6e0b\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-dj7tc"
Sep 18 19:43:56.264309 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:43:56.264255    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/80d81101-236a-4598-9ae0-f75d5dcb6e0b-certificates\") pod \"keda-metrics-apiserver-77857b7b98-dj7tc\" (UID: \"80d81101-236a-4598-9ae0-f75d5dcb6e0b\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-dj7tc"
Sep 18 19:43:56.264829 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:43:56.264346    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmzgq\" (UniqueName: \"kubernetes.io/projected/80d81101-236a-4598-9ae0-f75d5dcb6e0b-kube-api-access-xmzgq\") pod \"keda-metrics-apiserver-77857b7b98-dj7tc\" (UID: \"80d81101-236a-4598-9ae0-f75d5dcb6e0b\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-dj7tc"
Sep 18 19:43:56.264829 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:43:56.264405    2564 secret.go:281] references non-existent secret key: tls.crt
Sep 18 19:43:56.264829 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:43:56.264428    2564 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Sep 18 19:43:56.264829 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:43:56.264451    2564 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-77857b7b98-dj7tc: references non-existent secret key: tls.crt
Sep 18 19:43:56.264829 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:43:56.264537    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/80d81101-236a-4598-9ae0-f75d5dcb6e0b-certificates podName:80d81101-236a-4598-9ae0-f75d5dcb6e0b nodeName:}" failed. No retries permitted until 2026-09-18 19:43:56.764515285 +0000 UTC m=+353.274871358 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/80d81101-236a-4598-9ae0-f75d5dcb6e0b-certificates") pod "keda-metrics-apiserver-77857b7b98-dj7tc" (UID: "80d81101-236a-4598-9ae0-f75d5dcb6e0b") : references non-existent secret key: tls.crt
Sep 18 19:43:56.264829 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:43:56.264718    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/80d81101-236a-4598-9ae0-f75d5dcb6e0b-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-dj7tc\" (UID: \"80d81101-236a-4598-9ae0-f75d5dcb6e0b\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-dj7tc"
Sep 18 19:43:56.272898 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:43:56.272862    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmzgq\" (UniqueName: \"kubernetes.io/projected/80d81101-236a-4598-9ae0-f75d5dcb6e0b-kube-api-access-xmzgq\") pod \"keda-metrics-apiserver-77857b7b98-dj7tc\" (UID: \"80d81101-236a-4598-9ae0-f75d5dcb6e0b\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-dj7tc"
Sep 18 19:43:56.631702 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:43:56.631659    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:56.728340 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:43:56.728301    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:56.768551 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:43:56.768510    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/80d81101-236a-4598-9ae0-f75d5dcb6e0b-certificates\") pod \"keda-metrics-apiserver-77857b7b98-dj7tc\" (UID: \"80d81101-236a-4598-9ae0-f75d5dcb6e0b\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-dj7tc"
Sep 18 19:43:56.771167 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:43:56.771142    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/80d81101-236a-4598-9ae0-f75d5dcb6e0b-certificates\") pod \"keda-metrics-apiserver-77857b7b98-dj7tc\" (UID: \"80d81101-236a-4598-9ae0-f75d5dcb6e0b\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-dj7tc"
Sep 18 19:43:56.827732 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:43:56.827682    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:56.928810 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:43:56.928727    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:57.008391 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:43:57.008348    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-dj7tc"
Sep 18 19:43:57.029848 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:43:57.029813    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:57.128453 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:43:57.128416    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:57.288226 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:43:57.288122    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-dj7tc"]
Sep 18 19:43:57.291601 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:43:57.291570    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80d81101_236a_4598_9ae0_f75d5dcb6e0b.slice/crio-a16615c4dd47494c4758a4503ca6003d5d23b290b6181d7f063b5d79ee5c50db WatchSource:0}: Error finding container a16615c4dd47494c4758a4503ca6003d5d23b290b6181d7f063b5d79ee5c50db: Status 404 returned error can't find the container with id a16615c4dd47494c4758a4503ca6003d5d23b290b6181d7f063b5d79ee5c50db
Sep 18 19:43:57.291601 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:43:57.291575    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:57.353112 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:43:57.353063    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:57.654936 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:43:57.654901    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-dj7tc" event={"ID":"80d81101-236a-4598-9ae0-f75d5dcb6e0b","Type":"ContainerStarted","Data":"a16615c4dd47494c4758a4503ca6003d5d23b290b6181d7f063b5d79ee5c50db"}
Sep 18 19:44:00.670016 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:44:00.669954    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-dj7tc" event={"ID":"80d81101-236a-4598-9ae0-f75d5dcb6e0b","Type":"ContainerStarted","Data":"108927b461154a7383e2c91154683ee39ba35ae53a10ca5ca23bb3ebd9b3100c"}
Sep 18 19:44:00.670438 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:44:00.670134    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-dj7tc"
Sep 18 19:44:00.687570 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:44:00.687516    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-dj7tc" podStartSLOduration=1.8828351300000001 podStartE2EDuration="4.687492549s" totalImagesPullingTime="2.804657419s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:43:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:44:00.685791463 +0000 UTC m=+357.196147554" watchObservedRunningTime="2026-09-18 19:44:00.687492549 +0000 UTC m=+357.197848639"
Sep 18 19:44:11.680684 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:44:11.680656    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-dj7tc"
Sep 18 19:44:16.652220 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:44:16.652186    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-7q7km"
Sep 18 19:44:48.223579 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:44:48.223518    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:44:48.322437 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:44:48.322402    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:44:59.475694 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:44:59.475659    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:00.763712 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:00.763674    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:03.087887 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:03.087838    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:03.189587 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:03.189549    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:05.190517 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:05.190475    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:05.296819 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:05.296784    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:06.369598 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:06.369560    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-kkpzq"]
Sep 18 19:45:06.373198 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:06.373172    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-kkpzq"
Sep 18 19:45:06.375513 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:06.375480    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-z9pbf\""
Sep 18 19:45:06.375862 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:06.375843    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 18 19:45:06.375967 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:06.375877    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 18 19:45:06.383715 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:06.383689    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-kkpzq"]
Sep 18 19:45:06.473796 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:06.473751    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89scx\" (UniqueName: \"kubernetes.io/projected/9346d4b7-37b3-425e-893d-762f24742410-kube-api-access-89scx\") pod \"cert-manager-webhook-6486ffd997-kkpzq\" (UID: \"9346d4b7-37b3-425e-893d-762f24742410\") " pod="cert-manager/cert-manager-webhook-6486ffd997-kkpzq"
Sep 18 19:45:06.473971 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:06.473811    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9346d4b7-37b3-425e-893d-762f24742410-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-kkpzq\" (UID: \"9346d4b7-37b3-425e-893d-762f24742410\") " pod="cert-manager/cert-manager-webhook-6486ffd997-kkpzq"
Sep 18 19:45:06.575062 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:06.575020    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89scx\" (UniqueName: \"kubernetes.io/projected/9346d4b7-37b3-425e-893d-762f24742410-kube-api-access-89scx\") pod \"cert-manager-webhook-6486ffd997-kkpzq\" (UID: \"9346d4b7-37b3-425e-893d-762f24742410\") " pod="cert-manager/cert-manager-webhook-6486ffd997-kkpzq"
Sep 18 19:45:06.575240 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:06.575076    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9346d4b7-37b3-425e-893d-762f24742410-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-kkpzq\" (UID: \"9346d4b7-37b3-425e-893d-762f24742410\") " pod="cert-manager/cert-manager-webhook-6486ffd997-kkpzq"
Sep 18 19:45:06.582807 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:06.582767    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9346d4b7-37b3-425e-893d-762f24742410-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-kkpzq\" (UID: \"9346d4b7-37b3-425e-893d-762f24742410\") " pod="cert-manager/cert-manager-webhook-6486ffd997-kkpzq"
Sep 18 19:45:06.582957 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:06.582867    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89scx\" (UniqueName: \"kubernetes.io/projected/9346d4b7-37b3-425e-893d-762f24742410-kube-api-access-89scx\") pod \"cert-manager-webhook-6486ffd997-kkpzq\" (UID: \"9346d4b7-37b3-425e-893d-762f24742410\") " pod="cert-manager/cert-manager-webhook-6486ffd997-kkpzq"
Sep 18 19:45:06.698625 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:06.698585    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-kkpzq"
Sep 18 19:45:06.987045 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:06.986961    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-kkpzq"]
Sep 18 19:45:06.990089 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:45:06.990054    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9346d4b7_37b3_425e_893d_762f24742410.slice/crio-24457095aacc86df9f5f08895ea682471e13a797669a54d311abc24cdc6ab291 WatchSource:0}: Error finding container 24457095aacc86df9f5f08895ea682471e13a797669a54d311abc24cdc6ab291: Status 404 returned error can't find the container with id 24457095aacc86df9f5f08895ea682471e13a797669a54d311abc24cdc6ab291
Sep 18 19:45:07.328347 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:07.328261    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:07.430882 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:07.430837    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:07.529640 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:07.529601    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:07.632156 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:07.632120    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:07.729449 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:07.729412    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:07.830477 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:07.830435    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:07.929436 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:07.929340    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:07.964388 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:07.964353    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-kkpzq" event={"ID":"9346d4b7-37b3-425e-893d-762f24742410","Type":"ContainerStarted","Data":"24457095aacc86df9f5f08895ea682471e13a797669a54d311abc24cdc6ab291"}
Sep 18 19:45:08.061203 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:08.061102    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:08.160666 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:08.160607    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:09.588926 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:09.588871    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:09.688924 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:09.688890    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:09.789258 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:09.789212    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:10.182315 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:10.182273    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:10.980406 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:10.980374    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-kkpzq" event={"ID":"9346d4b7-37b3-425e-893d-762f24742410","Type":"ContainerStarted","Data":"80a2f036b2edd8903fa176f95364b65690144817569b3124276e462dba41d948"}
Sep 18 19:45:10.980783 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:10.980437    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-kkpzq"
Sep 18 19:45:10.995622 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:10.995574    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-kkpzq" podStartSLOduration=1.558006642 podStartE2EDuration="4.995560039s" totalImagesPullingTime="3.437553397s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:45:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:45:10.995086322 +0000 UTC m=+427.505442416" watchObservedRunningTime="2026-09-18 19:45:10.995560039 +0000 UTC m=+427.505916129"
Sep 18 19:45:12.140038 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:12.139970    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:15.194202 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:15.194167    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:16.986977 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:16.986941    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-kkpzq"
Sep 18 19:45:18.275116 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:18.275077    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:18.394550 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:18.394512    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:18.472050 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:18.472014    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b597c4d4f-jnjz9"]
Sep 18 19:45:18.586447 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:18.586369    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:18.684782 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:18.684746    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:18.784442 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:18.784413    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:21.688225 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:21.688189    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:21.789652 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:21.789614    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:33.337299 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:33.337207    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:37.746061 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:37.746023    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:37.758218 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:37.758184    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nkzvl"]
Sep 18 19:45:37.761484 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:37.761463    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nkzvl"
Sep 18 19:45:37.763372 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:37.763339    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 18 19:45:37.763372 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:37.763342    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 18 19:45:37.763619 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:37.763403    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 18 19:45:37.763667 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:37.763640    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-v9sgc\""
Sep 18 19:45:37.763735 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:37.763719    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 18 19:45:37.773297 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:37.773272    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nkzvl"]
Sep 18 19:45:37.815899 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:37.815869    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:37.842120 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:37.842085    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cea31a9b-a004-469c-904e-e68f8914fcc8-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-nkzvl\" (UID: \"cea31a9b-a004-469c-904e-e68f8914fcc8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nkzvl"
Sep 18 19:45:37.842120 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:37.842118    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cea31a9b-a004-469c-904e-e68f8914fcc8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-nkzvl\" (UID: \"cea31a9b-a004-469c-904e-e68f8914fcc8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nkzvl"
Sep 18 19:45:37.842370 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:37.842153    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cea31a9b-a004-469c-904e-e68f8914fcc8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-nkzvl\" (UID: \"cea31a9b-a004-469c-904e-e68f8914fcc8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nkzvl"
Sep 18 19:45:37.842370 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:37.842229    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5c2x\" (UniqueName: \"kubernetes.io/projected/cea31a9b-a004-469c-904e-e68f8914fcc8-kube-api-access-l5c2x\") pod \"istiod-openshift-gateway-7b7995d5cd-nkzvl\" (UID: \"cea31a9b-a004-469c-904e-e68f8914fcc8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nkzvl"
Sep 18 19:45:37.842370 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:37.842274    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cea31a9b-a004-469c-904e-e68f8914fcc8-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-nkzvl\" (UID: \"cea31a9b-a004-469c-904e-e68f8914fcc8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nkzvl"
Sep 18 19:45:37.842370 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:37.842338    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cea31a9b-a004-469c-904e-e68f8914fcc8-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-nkzvl\" (UID: \"cea31a9b-a004-469c-904e-e68f8914fcc8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nkzvl"
Sep 18 19:45:37.842522 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:37.842370    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cea31a9b-a004-469c-904e-e68f8914fcc8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-nkzvl\" (UID: \"cea31a9b-a004-469c-904e-e68f8914fcc8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nkzvl"
Sep 18 19:45:37.943501 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:37.943464    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cea31a9b-a004-469c-904e-e68f8914fcc8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-nkzvl\" (UID: \"cea31a9b-a004-469c-904e-e68f8914fcc8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nkzvl"
Sep 18 19:45:37.943501 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:37.943499    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5c2x\" (UniqueName: \"kubernetes.io/projected/cea31a9b-a004-469c-904e-e68f8914fcc8-kube-api-access-l5c2x\") pod \"istiod-openshift-gateway-7b7995d5cd-nkzvl\" (UID: \"cea31a9b-a004-469c-904e-e68f8914fcc8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nkzvl"
Sep 18 19:45:37.943727 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:37.943524    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cea31a9b-a004-469c-904e-e68f8914fcc8-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-nkzvl\" (UID: \"cea31a9b-a004-469c-904e-e68f8914fcc8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nkzvl"
Sep 18 19:45:37.943727 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:37.943567    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cea31a9b-a004-469c-904e-e68f8914fcc8-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-nkzvl\" (UID: \"cea31a9b-a004-469c-904e-e68f8914fcc8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nkzvl"
Sep 18 19:45:37.943727 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:37.943584    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cea31a9b-a004-469c-904e-e68f8914fcc8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-nkzvl\" (UID: \"cea31a9b-a004-469c-904e-e68f8914fcc8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nkzvl"
Sep 18 19:45:37.943727 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:37.943610    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cea31a9b-a004-469c-904e-e68f8914fcc8-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-nkzvl\" (UID: \"cea31a9b-a004-469c-904e-e68f8914fcc8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nkzvl"
Sep 18 19:45:37.943727 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:37.943624    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cea31a9b-a004-469c-904e-e68f8914fcc8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-nkzvl\" (UID: \"cea31a9b-a004-469c-904e-e68f8914fcc8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nkzvl"
Sep 18 19:45:37.944354 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:37.944330    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cea31a9b-a004-469c-904e-e68f8914fcc8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-nkzvl\" (UID: \"cea31a9b-a004-469c-904e-e68f8914fcc8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nkzvl"
Sep 18 19:45:37.946135 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:37.946115    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cea31a9b-a004-469c-904e-e68f8914fcc8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-nkzvl\" (UID: \"cea31a9b-a004-469c-904e-e68f8914fcc8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nkzvl"
Sep 18 19:45:37.946261 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:37.946245    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cea31a9b-a004-469c-904e-e68f8914fcc8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-nkzvl\" (UID: \"cea31a9b-a004-469c-904e-e68f8914fcc8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nkzvl"
Sep 18 19:45:37.946441 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:37.946417    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cea31a9b-a004-469c-904e-e68f8914fcc8-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-nkzvl\" (UID: \"cea31a9b-a004-469c-904e-e68f8914fcc8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nkzvl"
Sep 18 19:45:37.946476 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:37.946450    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cea31a9b-a004-469c-904e-e68f8914fcc8-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-nkzvl\" (UID: \"cea31a9b-a004-469c-904e-e68f8914fcc8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nkzvl"
Sep 18 19:45:37.950934 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:37.950907    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5c2x\" (UniqueName: \"kubernetes.io/projected/cea31a9b-a004-469c-904e-e68f8914fcc8-kube-api-access-l5c2x\") pod \"istiod-openshift-gateway-7b7995d5cd-nkzvl\" (UID: \"cea31a9b-a004-469c-904e-e68f8914fcc8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nkzvl"
Sep 18 19:45:37.951303 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:37.951282    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cea31a9b-a004-469c-904e-e68f8914fcc8-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-nkzvl\" (UID: \"cea31a9b-a004-469c-904e-e68f8914fcc8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nkzvl"
Sep 18 19:45:38.071676 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:38.071581    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nkzvl"
Sep 18 19:45:38.350797 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:38.350695    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nkzvl"]
Sep 18 19:45:38.354920 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:45:38.354888    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcea31a9b_a004_469c_904e_e68f8914fcc8.slice/crio-6f8698d384d88831d3044ef339a33c140c846f3b1ba8bbe18ee0383874837cf9 WatchSource:0}: Error finding container 6f8698d384d88831d3044ef339a33c140c846f3b1ba8bbe18ee0383874837cf9: Status 404 returned error can't find the container with id 6f8698d384d88831d3044ef339a33c140c846f3b1ba8bbe18ee0383874837cf9
Sep 18 19:45:38.640510 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:38.640476    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:39.097459 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:39.097374    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nkzvl" event={"ID":"cea31a9b-a004-469c-904e-e68f8914fcc8","Type":"ContainerStarted","Data":"6f8698d384d88831d3044ef339a33c140c846f3b1ba8bbe18ee0383874837cf9"}
Sep 18 19:45:40.944994 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:40.944942    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 18 19:45:40.945296 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:40.945021    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 18 19:45:41.108437 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:41.108401    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nkzvl" event={"ID":"cea31a9b-a004-469c-904e-e68f8914fcc8","Type":"ContainerStarted","Data":"960e44928654b7cea64634ac2dc5fc78d063c86d5e8fcc3ad50a457e9379a223"}
Sep 18 19:45:41.108635 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:41.108531    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nkzvl"
Sep 18 19:45:41.130875 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:41.130816    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nkzvl" podStartSLOduration=1.543139929 podStartE2EDuration="4.130797155s" totalImagesPullingTime="2.587657226s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:45:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:45:41.128829518 +0000 UTC m=+457.639185632" watchObservedRunningTime="2026-09-18 19:45:41.130797155 +0000 UTC m=+457.641153246"
Sep 18 19:45:41.678631 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:41.678583    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:41.777378 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:41.777346    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:42.114809 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:42.114729    2564 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-nkzvl container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 18 19:45:42.114809 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:42.114792    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nkzvl" podUID="cea31a9b-a004-469c-904e-e68f8914fcc8" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:45:43.495733 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:43.495684    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7b597c4d4f-jnjz9" podUID="f36fcc01-5634-4f3c-8c82-3b982107d739" containerName="console" containerID="cri-o://14de7e9ff7284e6adfb74d4432496fa0a06aeba37068fd98cd0fcb39a2a04588" gracePeriod=15
Sep 18 19:45:43.553129 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:43.553103    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:43.640437 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:43.640402    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:43.780365 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:43.780330    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b597c4d4f-jnjz9"
Sep 18 19:45:43.897232 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:43.897195    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f36fcc01-5634-4f3c-8c82-3b982107d739-trusted-ca-bundle\") pod \"f36fcc01-5634-4f3c-8c82-3b982107d739\" (UID: \"f36fcc01-5634-4f3c-8c82-3b982107d739\") "
Sep 18 19:45:43.897412 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:43.897266    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f36fcc01-5634-4f3c-8c82-3b982107d739-console-serving-cert\") pod \"f36fcc01-5634-4f3c-8c82-3b982107d739\" (UID: \"f36fcc01-5634-4f3c-8c82-3b982107d739\") "
Sep 18 19:45:43.897412 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:43.897296    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f36fcc01-5634-4f3c-8c82-3b982107d739-oauth-serving-cert\") pod \"f36fcc01-5634-4f3c-8c82-3b982107d739\" (UID: \"f36fcc01-5634-4f3c-8c82-3b982107d739\") "
Sep 18 19:45:43.897412 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:43.897350    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f36fcc01-5634-4f3c-8c82-3b982107d739-console-config\") pod \"f36fcc01-5634-4f3c-8c82-3b982107d739\" (UID: \"f36fcc01-5634-4f3c-8c82-3b982107d739\") "
Sep 18 19:45:43.897412 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:43.897380    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bch58\" (UniqueName: \"kubernetes.io/projected/f36fcc01-5634-4f3c-8c82-3b982107d739-kube-api-access-bch58\") pod \"f36fcc01-5634-4f3c-8c82-3b982107d739\" (UID: \"f36fcc01-5634-4f3c-8c82-3b982107d739\") "
Sep 18 19:45:43.897608 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:43.897413    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f36fcc01-5634-4f3c-8c82-3b982107d739-service-ca\") pod \"f36fcc01-5634-4f3c-8c82-3b982107d739\" (UID: \"f36fcc01-5634-4f3c-8c82-3b982107d739\") "
Sep 18 19:45:43.897608 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:43.897454    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f36fcc01-5634-4f3c-8c82-3b982107d739-console-oauth-config\") pod \"f36fcc01-5634-4f3c-8c82-3b982107d739\" (UID: \"f36fcc01-5634-4f3c-8c82-3b982107d739\") "
Sep 18 19:45:43.897912 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:43.897777    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f36fcc01-5634-4f3c-8c82-3b982107d739-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f36fcc01-5634-4f3c-8c82-3b982107d739" (UID: "f36fcc01-5634-4f3c-8c82-3b982107d739"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:45:43.897912 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:43.897804    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f36fcc01-5634-4f3c-8c82-3b982107d739-console-config" (OuterVolumeSpecName: "console-config") pod "f36fcc01-5634-4f3c-8c82-3b982107d739" (UID: "f36fcc01-5634-4f3c-8c82-3b982107d739"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:45:43.897912 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:43.897695    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f36fcc01-5634-4f3c-8c82-3b982107d739-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "f36fcc01-5634-4f3c-8c82-3b982107d739" (UID: "f36fcc01-5634-4f3c-8c82-3b982107d739"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:45:43.897912 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:43.897875    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f36fcc01-5634-4f3c-8c82-3b982107d739-service-ca" (OuterVolumeSpecName: "service-ca") pod "f36fcc01-5634-4f3c-8c82-3b982107d739" (UID: "f36fcc01-5634-4f3c-8c82-3b982107d739"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:45:43.899710 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:43.899680    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f36fcc01-5634-4f3c-8c82-3b982107d739-kube-api-access-bch58" (OuterVolumeSpecName: "kube-api-access-bch58") pod "f36fcc01-5634-4f3c-8c82-3b982107d739" (UID: "f36fcc01-5634-4f3c-8c82-3b982107d739"). InnerVolumeSpecName "kube-api-access-bch58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:45:43.899710 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:43.899688    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f36fcc01-5634-4f3c-8c82-3b982107d739-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f36fcc01-5634-4f3c-8c82-3b982107d739" (UID: "f36fcc01-5634-4f3c-8c82-3b982107d739"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:45:43.899854 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:43.899751    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f36fcc01-5634-4f3c-8c82-3b982107d739-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f36fcc01-5634-4f3c-8c82-3b982107d739" (UID: "f36fcc01-5634-4f3c-8c82-3b982107d739"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:45:43.998459 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:43.998422    2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f36fcc01-5634-4f3c-8c82-3b982107d739-console-config\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 19:45:43.998459 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:43.998455    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bch58\" (UniqueName: \"kubernetes.io/projected/f36fcc01-5634-4f3c-8c82-3b982107d739-kube-api-access-bch58\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 19:45:43.998667 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:43.998471    2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f36fcc01-5634-4f3c-8c82-3b982107d739-service-ca\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 19:45:43.998667 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:43.998485    2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f36fcc01-5634-4f3c-8c82-3b982107d739-console-oauth-config\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 19:45:43.998667 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:43.998502    2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f36fcc01-5634-4f3c-8c82-3b982107d739-trusted-ca-bundle\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 19:45:43.998667 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:43.998518    2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f36fcc01-5634-4f3c-8c82-3b982107d739-console-serving-cert\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 19:45:43.998667 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:43.998535    2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f36fcc01-5634-4f3c-8c82-3b982107d739-oauth-serving-cert\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 19:45:44.123111 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:44.123080    2564 generic.go:356] "Generic (PLEG): container finished" podID="f36fcc01-5634-4f3c-8c82-3b982107d739" containerID="14de7e9ff7284e6adfb74d4432496fa0a06aeba37068fd98cd0fcb39a2a04588" exitCode=0
Sep 18 19:45:44.123286 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:44.123126    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b597c4d4f-jnjz9"
Sep 18 19:45:44.123286 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:44.123169    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b597c4d4f-jnjz9" event={"ID":"f36fcc01-5634-4f3c-8c82-3b982107d739","Type":"ContainerDied","Data":"14de7e9ff7284e6adfb74d4432496fa0a06aeba37068fd98cd0fcb39a2a04588"}
Sep 18 19:45:44.123286 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:44.123219    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b597c4d4f-jnjz9" event={"ID":"f36fcc01-5634-4f3c-8c82-3b982107d739","Type":"ContainerDied","Data":"b4de2e14ad3547664ac6c4df1ba0a12c6238c135f54279dd85fbf3685cbcaa54"}
Sep 18 19:45:44.123286 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:44.123240    2564 scope.go:117] "RemoveContainer" containerID="14de7e9ff7284e6adfb74d4432496fa0a06aeba37068fd98cd0fcb39a2a04588"
Sep 18 19:45:44.131646 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:44.131622    2564 scope.go:117] "RemoveContainer" containerID="14de7e9ff7284e6adfb74d4432496fa0a06aeba37068fd98cd0fcb39a2a04588"
Sep 18 19:45:44.131968 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:45:44.131946    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14de7e9ff7284e6adfb74d4432496fa0a06aeba37068fd98cd0fcb39a2a04588\": container with ID starting with 14de7e9ff7284e6adfb74d4432496fa0a06aeba37068fd98cd0fcb39a2a04588 not found: ID does not exist" containerID="14de7e9ff7284e6adfb74d4432496fa0a06aeba37068fd98cd0fcb39a2a04588"
Sep 18 19:45:44.132049 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:44.131992    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14de7e9ff7284e6adfb74d4432496fa0a06aeba37068fd98cd0fcb39a2a04588"} err="failed to get container status \"14de7e9ff7284e6adfb74d4432496fa0a06aeba37068fd98cd0fcb39a2a04588\": rpc error: code = NotFound desc = could not find container \"14de7e9ff7284e6adfb74d4432496fa0a06aeba37068fd98cd0fcb39a2a04588\": container with ID starting with 14de7e9ff7284e6adfb74d4432496fa0a06aeba37068fd98cd0fcb39a2a04588 not found: ID does not exist"
Sep 18 19:45:44.138228 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:44.138205    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b597c4d4f-jnjz9"]
Sep 18 19:45:44.144660 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:44.144635    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7b597c4d4f-jnjz9"]
Sep 18 19:45:45.113791 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:45.113761    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nkzvl"
Sep 18 19:45:45.581238 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:45.581200    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-ggzph"]
Sep 18 19:45:45.581575 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:45.581560    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f36fcc01-5634-4f3c-8c82-3b982107d739" containerName="console"
Sep 18 19:45:45.581637 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:45.581577    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f36fcc01-5634-4f3c-8c82-3b982107d739" containerName="console"
Sep 18 19:45:45.581637 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:45.581632    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f36fcc01-5634-4f3c-8c82-3b982107d739" containerName="console"
Sep 18 19:45:45.584299 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:45.584274    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-ggzph"
Sep 18 19:45:45.586392 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:45.586365    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-2jvmk\""
Sep 18 19:45:45.599711 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:45.599678    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-ggzph"]
Sep 18 19:45:45.624092 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:45.624055    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:45.693858 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:45.693825    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:45.714593 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:45.714558    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1c0f546-0824-446c-8f25-ffed46184e1c-istio-envoy\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-ggzph\" (UID: \"a1c0f546-0824-446c-8f25-ffed46184e1c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-ggzph"
Sep 18 19:45:45.714786 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:45.714604    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1c0f546-0824-446c-8f25-ffed46184e1c-istio-data\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-ggzph\" (UID: \"a1c0f546-0824-446c-8f25-ffed46184e1c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-ggzph"
Sep 18 19:45:45.714786 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:45.714694    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1c0f546-0824-446c-8f25-ffed46184e1c-istio-token\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-ggzph\" (UID: \"a1c0f546-0824-446c-8f25-ffed46184e1c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-ggzph"
Sep 18 19:45:45.714786 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:45.714744    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbnbn\" (UniqueName: \"kubernetes.io/projected/a1c0f546-0824-446c-8f25-ffed46184e1c-kube-api-access-wbnbn\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-ggzph\" (UID: \"a1c0f546-0824-446c-8f25-ffed46184e1c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-ggzph"
Sep 18 19:45:45.714786 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:45.714784    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c0f546-0824-446c-8f25-ffed46184e1c-credential-socket\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-ggzph\" (UID: \"a1c0f546-0824-446c-8f25-ffed46184e1c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-ggzph"
Sep 18 19:45:45.714963 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:45.714809    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1c0f546-0824-446c-8f25-ffed46184e1c-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-ggzph\" (UID: \"a1c0f546-0824-446c-8f25-ffed46184e1c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-ggzph"
Sep 18 19:45:45.714963 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:45.714874    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1c0f546-0824-446c-8f25-ffed46184e1c-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-ggzph\" (UID: \"a1c0f546-0824-446c-8f25-ffed46184e1c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-ggzph"
Sep 18 19:45:45.714963 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:45.714905    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1c0f546-0824-446c-8f25-ffed46184e1c-workload-certs\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-ggzph\" (UID: \"a1c0f546-0824-446c-8f25-ffed46184e1c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-ggzph"
Sep 18 19:45:45.714963 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:45.714924    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c0f546-0824-446c-8f25-ffed46184e1c-workload-socket\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-ggzph\" (UID: \"a1c0f546-0824-446c-8f25-ffed46184e1c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-ggzph"
Sep 18 19:45:45.815569 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:45.815526    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1c0f546-0824-446c-8f25-ffed46184e1c-istio-data\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-ggzph\" (UID: \"a1c0f546-0824-446c-8f25-ffed46184e1c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-ggzph"
Sep 18 19:45:45.815569 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:45.815579    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1c0f546-0824-446c-8f25-ffed46184e1c-istio-token\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-ggzph\" (UID: \"a1c0f546-0824-446c-8f25-ffed46184e1c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-ggzph"
Sep 18 19:45:45.815843 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:45.815693    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbnbn\" (UniqueName: \"kubernetes.io/projected/a1c0f546-0824-446c-8f25-ffed46184e1c-kube-api-access-wbnbn\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-ggzph\" (UID: \"a1c0f546-0824-446c-8f25-ffed46184e1c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-ggzph"
Sep 18 19:45:45.815843 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:45.815739    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c0f546-0824-446c-8f25-ffed46184e1c-credential-socket\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-ggzph\" (UID: \"a1c0f546-0824-446c-8f25-ffed46184e1c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-ggzph"
Sep 18 19:45:45.815843 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:45.815759    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1c0f546-0824-446c-8f25-ffed46184e1c-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-ggzph\" (UID: \"a1c0f546-0824-446c-8f25-ffed46184e1c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-ggzph"
Sep 18 19:45:45.815843 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:45.815793    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1c0f546-0824-446c-8f25-ffed46184e1c-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-ggzph\" (UID: \"a1c0f546-0824-446c-8f25-ffed46184e1c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-ggzph"
Sep 18 19:45:45.815843 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:45.815813    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1c0f546-0824-446c-8f25-ffed46184e1c-workload-certs\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-ggzph\" (UID: \"a1c0f546-0824-446c-8f25-ffed46184e1c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-ggzph"
Sep 18 19:45:45.815843 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:45.815836    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c0f546-0824-446c-8f25-ffed46184e1c-workload-socket\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-ggzph\" (UID: \"a1c0f546-0824-446c-8f25-ffed46184e1c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-ggzph"
Sep 18 19:45:45.816142 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:45.815918    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1c0f546-0824-446c-8f25-ffed46184e1c-istio-envoy\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-ggzph\" (UID: \"a1c0f546-0824-446c-8f25-ffed46184e1c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-ggzph"
Sep 18 19:45:45.816142 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:45.816008    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1c0f546-0824-446c-8f25-ffed46184e1c-istio-data\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-ggzph\" (UID: \"a1c0f546-0824-446c-8f25-ffed46184e1c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-ggzph"
Sep 18 19:45:45.816209 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:45.816193    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c0f546-0824-446c-8f25-ffed46184e1c-credential-socket\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-ggzph\" (UID: \"a1c0f546-0824-446c-8f25-ffed46184e1c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-ggzph"
Sep 18 19:45:45.816273 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:45.816255    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1c0f546-0824-446c-8f25-ffed46184e1c-workload-certs\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-ggzph\" (UID: \"a1c0f546-0824-446c-8f25-ffed46184e1c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-ggzph"
Sep 18 19:45:45.816447 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:45.816414    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c0f546-0824-446c-8f25-ffed46184e1c-workload-socket\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-ggzph\" (UID: \"a1c0f546-0824-446c-8f25-ffed46184e1c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-ggzph"
Sep 18 19:45:45.816577 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:45.816474    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1c0f546-0824-446c-8f25-ffed46184e1c-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-ggzph\" (UID: \"a1c0f546-0824-446c-8f25-ffed46184e1c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-ggzph"
Sep 18 19:45:45.818322 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:45.818301    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1c0f546-0824-446c-8f25-ffed46184e1c-istio-envoy\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-ggzph\" (UID: \"a1c0f546-0824-446c-8f25-ffed46184e1c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-ggzph"
Sep 18 19:45:45.818462 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:45.818443    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1c0f546-0824-446c-8f25-ffed46184e1c-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-ggzph\" (UID: \"a1c0f546-0824-446c-8f25-ffed46184e1c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-ggzph"
Sep 18 19:45:45.822640 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:45.822614    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1c0f546-0824-446c-8f25-ffed46184e1c-istio-token\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-ggzph\" (UID: \"a1c0f546-0824-446c-8f25-ffed46184e1c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-ggzph"
Sep 18 19:45:45.822753 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:45.822740    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbnbn\" (UniqueName: \"kubernetes.io/projected/a1c0f546-0824-446c-8f25-ffed46184e1c-kube-api-access-wbnbn\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-ggzph\" (UID: \"a1c0f546-0824-446c-8f25-ffed46184e1c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-ggzph"
Sep 18 19:45:45.899672 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:45.899629    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-ggzph"
Sep 18 19:45:45.999451 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:45.999413    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f36fcc01-5634-4f3c-8c82-3b982107d739" path="/var/lib/kubelet/pods/f36fcc01-5634-4f3c-8c82-3b982107d739/volumes"
Sep 18 19:45:46.175182 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:46.175019    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-ggzph"]
Sep 18 19:45:46.177951 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:45:46.177923    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1c0f546_0824_446c_8f25_ffed46184e1c.slice/crio-b6f24316c40046426e79887aff58191bc09d98354d5147c6170585a3b21a84f5 WatchSource:0}: Error finding container b6f24316c40046426e79887aff58191bc09d98354d5147c6170585a3b21a84f5: Status 404 returned error can't find the container with id b6f24316c40046426e79887aff58191bc09d98354d5147c6170585a3b21a84f5
Sep 18 19:45:47.140165 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:47.140125    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-ggzph" event={"ID":"a1c0f546-0824-446c-8f25-ffed46184e1c","Type":"ContainerStarted","Data":"b6f24316c40046426e79887aff58191bc09d98354d5147c6170585a3b21a84f5"}
Sep 18 19:45:48.431358 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:48.431270    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:48.697152 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:48.697119    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 18 19:45:48.697228 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:48.697189    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 18 19:45:48.697228 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:48.697221    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 18 19:45:49.151099 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:49.151067    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-ggzph" event={"ID":"a1c0f546-0824-446c-8f25-ffed46184e1c","Type":"ContainerStarted","Data":"8620bf11b83c057f761dfd87a965aefcf57d9958b869f2c6224ad25594c1b70b"}
Sep 18 19:45:49.171474 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:49.171421    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-ggzph" podStartSLOduration=1.6547066350000001 podStartE2EDuration="4.171403558s" totalImagesPullingTime="2.516696923s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:45:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:45:49.169530964 +0000 UTC m=+465.679887083" watchObservedRunningTime="2026-09-18 19:45:49.171403558 +0000 UTC m=+465.681759640"
Sep 18 19:45:49.900341 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:49.900308    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-ggzph"
Sep 18 19:45:49.904902 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:49.904877    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-ggzph"
Sep 18 19:45:50.155539 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:50.155452    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-ggzph"
Sep 18 19:45:50.156378 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:50.156357    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-ggzph"
Sep 18 19:45:51.093679 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:51.093644    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:51.202954 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:45:51.202912    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:46:01.923045 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:01.923006    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:46:03.975858 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:03.975809    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65dc8f76f5-8bfn7"]
Sep 18 19:46:03.979093 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:03.979076    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-8bfn7"
Sep 18 19:46:03.983664 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:03.983638    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 18 19:46:03.983664 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:03.983657    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-w6292\""
Sep 18 19:46:03.983835 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:03.983793    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 18 19:46:03.983890 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:03.983876    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 18 19:46:03.993688 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:03.993659    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65dc8f76f5-8bfn7"]
Sep 18 19:46:04.071805 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:04.071758    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm7jb\" (UniqueName: \"kubernetes.io/projected/81c1bdbc-a919-4ce9-9c99-501c8250ea0e-kube-api-access-pm7jb\") pod \"dns-operator-controller-manager-65dc8f76f5-8bfn7\" (UID: \"81c1bdbc-a919-4ce9-9c99-501c8250ea0e\") " pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-8bfn7"
Sep 18 19:46:04.172946 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:04.172901    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm7jb\" (UniqueName: \"kubernetes.io/projected/81c1bdbc-a919-4ce9-9c99-501c8250ea0e-kube-api-access-pm7jb\") pod \"dns-operator-controller-manager-65dc8f76f5-8bfn7\" (UID: \"81c1bdbc-a919-4ce9-9c99-501c8250ea0e\") " pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-8bfn7"
Sep 18 19:46:04.181596 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:04.181565    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm7jb\" (UniqueName: \"kubernetes.io/projected/81c1bdbc-a919-4ce9-9c99-501c8250ea0e-kube-api-access-pm7jb\") pod \"dns-operator-controller-manager-65dc8f76f5-8bfn7\" (UID: \"81c1bdbc-a919-4ce9-9c99-501c8250ea0e\") " pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-8bfn7"
Sep 18 19:46:04.289450 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:04.289365    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-8bfn7"
Sep 18 19:46:04.306063 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:04.306027    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:46:04.570694 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:04.570605    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65dc8f76f5-8bfn7"]
Sep 18 19:46:04.573158 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:46:04.573130    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81c1bdbc_a919_4ce9_9c99_501c8250ea0e.slice/crio-179be9504faa0f3d644f42e411f5857f3e22553ec7fee7b4aaf439d927304887 WatchSource:0}: Error finding container 179be9504faa0f3d644f42e411f5857f3e22553ec7fee7b4aaf439d927304887: Status 404 returned error can't find the container with id 179be9504faa0f3d644f42e411f5857f3e22553ec7fee7b4aaf439d927304887
Sep 18 19:46:04.720784 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:04.720741    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:46:05.223029 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:05.222993    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-8bfn7" event={"ID":"81c1bdbc-a919-4ce9-9c99-501c8250ea0e","Type":"ContainerStarted","Data":"179be9504faa0f3d644f42e411f5857f3e22553ec7fee7b4aaf439d927304887"}
Sep 18 19:46:05.761043 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:05.761003    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:46:06.672027 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:06.671972    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-ksj7l"]
Sep 18 19:46:06.675735 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:06.675707    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-6fc8f864f4-ksj7l"
Sep 18 19:46:06.677957 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:06.677936    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-n9d5n\""
Sep 18 19:46:06.684953 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:06.684926    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-ksj7l"]
Sep 18 19:46:06.797533 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:06.797494    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55pt7\" (UniqueName: \"kubernetes.io/projected/2ee27ad9-6661-4074-9507-5fc815d026ca-kube-api-access-55pt7\") pod \"authorino-operator-6fc8f864f4-ksj7l\" (UID: \"2ee27ad9-6661-4074-9507-5fc815d026ca\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-ksj7l"
Sep 18 19:46:06.898766 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:06.898708    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55pt7\" (UniqueName: \"kubernetes.io/projected/2ee27ad9-6661-4074-9507-5fc815d026ca-kube-api-access-55pt7\") pod \"authorino-operator-6fc8f864f4-ksj7l\" (UID: \"2ee27ad9-6661-4074-9507-5fc815d026ca\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-ksj7l"
Sep 18 19:46:06.913951 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:06.913907    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55pt7\" (UniqueName: \"kubernetes.io/projected/2ee27ad9-6661-4074-9507-5fc815d026ca-kube-api-access-55pt7\") pod \"authorino-operator-6fc8f864f4-ksj7l\" (UID: \"2ee27ad9-6661-4074-9507-5fc815d026ca\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-ksj7l"
Sep 18 19:46:06.989461 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:06.989358    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-6fc8f864f4-ksj7l"
Sep 18 19:46:07.161397 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:07.161354    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:46:07.234153 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:07.234120    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-8bfn7" event={"ID":"81c1bdbc-a919-4ce9-9c99-501c8250ea0e","Type":"ContainerStarted","Data":"a18da0b90407076eef4ec432baf8513eb14e747114f18f3b42fedcb5a9b3e778"}
Sep 18 19:46:07.234334 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:07.234301    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-8bfn7"
Sep 18 19:46:07.254650 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:07.254549    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-8bfn7" podStartSLOduration=1.751757128 podStartE2EDuration="4.25452975s" totalImagesPullingTime="2.502772622s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:46:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:46:07.252721983 +0000 UTC m=+483.763078087" watchObservedRunningTime="2026-09-18 19:46:07.25452975 +0000 UTC m=+483.764885841"
Sep 18 19:46:07.287780 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:07.287752    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-ksj7l"]
Sep 18 19:46:07.290516 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:46:07.290483    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ee27ad9_6661_4074_9507_5fc815d026ca.slice/crio-3d52b6f4b2386cb3362c1dd08df65b504349e1c1da62461e37c9105f518475d7 WatchSource:0}: Error finding container 3d52b6f4b2386cb3362c1dd08df65b504349e1c1da62461e37c9105f518475d7: Status 404 returned error can't find the container with id 3d52b6f4b2386cb3362c1dd08df65b504349e1c1da62461e37c9105f518475d7
Sep 18 19:46:07.314021 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:07.313969    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:46:08.246528 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:08.246471    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-6fc8f864f4-ksj7l" event={"ID":"2ee27ad9-6661-4074-9507-5fc815d026ca","Type":"ContainerStarted","Data":"3d52b6f4b2386cb3362c1dd08df65b504349e1c1da62461e37c9105f518475d7"}
Sep 18 19:46:08.563033 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:08.562930    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:46:08.893112 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:08.893079    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-8rb5h"]
Sep 18 19:46:08.897790 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:08.897769    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-8rb5h"
Sep 18 19:46:08.899788 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:08.899765    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-j9r9t\""
Sep 18 19:46:08.910105 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:08.910078    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-8rb5h"]
Sep 18 19:46:09.018653 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:09.018612    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkltz\" (UniqueName: \"kubernetes.io/projected/107236fd-4036-4417-a437-f674c2402002-kube-api-access-lkltz\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-8rb5h\" (UID: \"107236fd-4036-4417-a437-f674c2402002\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-8rb5h"
Sep 18 19:46:09.119958 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:09.119918    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkltz\" (UniqueName: \"kubernetes.io/projected/107236fd-4036-4417-a437-f674c2402002-kube-api-access-lkltz\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-8rb5h\" (UID: \"107236fd-4036-4417-a437-f674c2402002\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-8rb5h"
Sep 18 19:46:09.129170 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:09.129145    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkltz\" (UniqueName: \"kubernetes.io/projected/107236fd-4036-4417-a437-f674c2402002-kube-api-access-lkltz\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-8rb5h\" (UID: \"107236fd-4036-4417-a437-f674c2402002\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-8rb5h"
Sep 18 19:46:09.207977 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:09.207882    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-8rb5h"
Sep 18 19:46:09.254058 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:09.253977    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-6fc8f864f4-ksj7l" event={"ID":"2ee27ad9-6661-4074-9507-5fc815d026ca","Type":"ContainerStarted","Data":"2d0b7eb30ae31b287d7c85868e60bfb42e61cd319f7f335f8a6a9ff204905efc"}
Sep 18 19:46:09.254557 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:09.254163    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-6fc8f864f4-ksj7l"
Sep 18 19:46:09.280543 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:09.280471    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-6fc8f864f4-ksj7l" podStartSLOduration=1.9979329460000002 podStartE2EDuration="3.280446215s" totalImagesPullingTime="1.282513269s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:46:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:46:09.278650233 +0000 UTC m=+485.789006357" watchObservedRunningTime="2026-09-18 19:46:09.280446215 +0000 UTC m=+485.790802306"
Sep 18 19:46:09.509323 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:09.509230    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-8rb5h"]
Sep 18 19:46:09.512913 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:46:09.512881    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod107236fd_4036_4417_a437_f674c2402002.slice/crio-becb9f2fd31bc68f5a9e0ed2c1d1d2f6f319d9af51e1ebb6cbc6f0c12f666a93 WatchSource:0}: Error finding container becb9f2fd31bc68f5a9e0ed2c1d1d2f6f319d9af51e1ebb6cbc6f0c12f666a93: Status 404 returned error can't find the container with id becb9f2fd31bc68f5a9e0ed2c1d1d2f6f319d9af51e1ebb6cbc6f0c12f666a93
Sep 18 19:46:09.548564 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:09.548521    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:46:09.648457 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:09.648398    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:46:09.915815 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:09.915777    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:46:10.024028 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:10.023949    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:46:10.261124 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:10.261030    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-8rb5h" event={"ID":"107236fd-4036-4417-a437-f674c2402002","Type":"ContainerStarted","Data":"becb9f2fd31bc68f5a9e0ed2c1d1d2f6f319d9af51e1ebb6cbc6f0c12f666a93"}
Sep 18 19:46:10.315623 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:10.315576    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:46:10.413745 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:10.413708    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:46:11.268199 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:11.268089    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-8rb5h" event={"ID":"107236fd-4036-4417-a437-f674c2402002","Type":"ContainerStarted","Data":"1da09f42f1107e6b648befb492a670c4dd9398afafe0d83834d1ecde71c67916"}
Sep 18 19:46:11.268199 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:11.268160    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-8rb5h"
Sep 18 19:46:11.283956 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:11.283903    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-8rb5h" podStartSLOduration=1.885520363 podStartE2EDuration="3.283885261s" totalImagesPullingTime="1.398364898s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:46:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:46:11.282572839 +0000 UTC m=+487.792928949" watchObservedRunningTime="2026-09-18 19:46:11.283885261 +0000 UTC m=+487.794241353"
Sep 18 19:46:15.329828 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:15.329785    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:46:15.426539 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:15.426500    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:46:18.249565 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:18.249537    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-8bfn7"
Sep 18 19:46:20.264453 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:20.264351    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-6fc8f864f4-ksj7l"
Sep 18 19:46:20.290053 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:20.290012    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:46:20.459084 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:20.459040    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:46:20.970732 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:20.970691    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:46:21.070407 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:21.070375    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:46:21.396086 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:21.396049    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:46:21.478481 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:21.478435    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:46:22.276044 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:22.276012    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-8rb5h"
Sep 18 19:46:22.949752 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:22.949718    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:46:23.051173 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:23.051133    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:46:27.678320 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:27.678280    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:46:27.772342 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:27.772306    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:46:27.834769 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:27.834730    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:46:34.362501 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:34.362461    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:46:34.463862 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:34.463819    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:46:39.755309 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:39.755271    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:46:39.813209 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:39.813172    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:46:40.160602 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:40.160558    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:46:40.213133 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:40.213094    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:46:40.802472 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:40.802412    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:46:43.558655 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:43.558617    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:46:43.659147 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:43.659104    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:46:47.307402 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:47.307370    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:46:47.405898 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:47.405852    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:46:47.465855 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:47.465809    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:46:52.029464 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:52.029422    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:46:52.091663 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:52.091624    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:46:54.636647 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:54.636609    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:46:54.738149 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:54.738109    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:46:57.829239 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:46:57.829151    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:47:03.607080 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:03.607037    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:47:03.613888 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:03.613855    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-bc6xr"]
Sep 18 19:47:03.617567 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:03.617546    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-bc6xr"
Sep 18 19:47:03.619618 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:03.619591    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 18 19:47:03.619739 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:03.619691    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-jkpnw\""
Sep 18 19:47:03.626671 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:03.626649    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-bc6xr"]
Sep 18 19:47:03.708488 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:03.708442    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-bc6xr"]
Sep 18 19:47:03.717147 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:03.717106    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:47:03.774592 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:03.774552    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:47:03.784768 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:03.784736    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e1919358-3891-4c29-a6ce-cfcde9675c53-config-file\") pod \"limitador-limitador-8588749977-bc6xr\" (UID: \"e1919358-3891-4c29-a6ce-cfcde9675c53\") " pod="kuadrant-system/limitador-limitador-8588749977-bc6xr"
Sep 18 19:47:03.784935 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:03.784800    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g887t\" (UniqueName: \"kubernetes.io/projected/e1919358-3891-4c29-a6ce-cfcde9675c53-kube-api-access-g887t\") pod \"limitador-limitador-8588749977-bc6xr\" (UID: \"e1919358-3891-4c29-a6ce-cfcde9675c53\") " pod="kuadrant-system/limitador-limitador-8588749977-bc6xr"
Sep 18 19:47:03.885828 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:03.885794    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e1919358-3891-4c29-a6ce-cfcde9675c53-config-file\") pod \"limitador-limitador-8588749977-bc6xr\" (UID: \"e1919358-3891-4c29-a6ce-cfcde9675c53\") " pod="kuadrant-system/limitador-limitador-8588749977-bc6xr"
Sep 18 19:47:03.886041 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:03.885858    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g887t\" (UniqueName: \"kubernetes.io/projected/e1919358-3891-4c29-a6ce-cfcde9675c53-kube-api-access-g887t\") pod \"limitador-limitador-8588749977-bc6xr\" (UID: \"e1919358-3891-4c29-a6ce-cfcde9675c53\") " pod="kuadrant-system/limitador-limitador-8588749977-bc6xr"
Sep 18 19:47:03.887870 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:03.887849    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 18 19:47:03.893652 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:03.893628    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g887t\" (UniqueName: \"kubernetes.io/projected/e1919358-3891-4c29-a6ce-cfcde9675c53-kube-api-access-g887t\") pod \"limitador-limitador-8588749977-bc6xr\" (UID: \"e1919358-3891-4c29-a6ce-cfcde9675c53\") " pod="kuadrant-system/limitador-limitador-8588749977-bc6xr"
Sep 18 19:47:03.896738 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:03.896717    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e1919358-3891-4c29-a6ce-cfcde9675c53-config-file\") pod \"limitador-limitador-8588749977-bc6xr\" (UID: \"e1919358-3891-4c29-a6ce-cfcde9675c53\") " pod="kuadrant-system/limitador-limitador-8588749977-bc6xr"
Sep 18 19:47:03.930429 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:03.930391    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-jkpnw\""
Sep 18 19:47:03.938860 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:03.938825    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-bc6xr"
Sep 18 19:47:04.211072 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:04.210953    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-bc6xr"]
Sep 18 19:47:04.213891 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:47:04.213867    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1919358_3891_4c29_a6ce_cfcde9675c53.slice/crio-4125bedba038321c9ddf2d03004fcd9c33b7686edb87490d65a094411b1652a3 WatchSource:0}: Error finding container 4125bedba038321c9ddf2d03004fcd9c33b7686edb87490d65a094411b1652a3: Status 404 returned error can't find the container with id 4125bedba038321c9ddf2d03004fcd9c33b7686edb87490d65a094411b1652a3
Sep 18 19:47:04.421859 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:04.421819    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:47:04.515941 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:04.515859    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-bc6xr" event={"ID":"e1919358-3891-4c29-a6ce-cfcde9675c53","Type":"ContainerStarted","Data":"4125bedba038321c9ddf2d03004fcd9c33b7686edb87490d65a094411b1652a3"}
Sep 18 19:47:04.533775 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:04.533744    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:47:06.528862 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:06.528821    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-bc6xr" event={"ID":"e1919358-3891-4c29-a6ce-cfcde9675c53","Type":"ContainerStarted","Data":"af0dab12d92e1f7f308f1d067a465603d8019a40a2f22dcd951b71ed735be1b0"}
Sep 18 19:47:06.529293 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:06.528948    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-bc6xr"
Sep 18 19:47:06.543204 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:06.543149    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-bc6xr" podStartSLOduration=2.043877867 podStartE2EDuration="3.543131923s" totalImagesPullingTime="1.499254056s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:47:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:47:06.541794426 +0000 UTC m=+543.052150534" watchObservedRunningTime="2026-09-18 19:47:06.543131923 +0000 UTC m=+543.053488013"
Sep 18 19:47:17.056004 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:17.051996    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-bc6xr"]
Sep 18 19:47:17.056004 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:17.052333    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-bc6xr" podUID="e1919358-3891-4c29-a6ce-cfcde9675c53" containerName="limitador" containerID="cri-o://af0dab12d92e1f7f308f1d067a465603d8019a40a2f22dcd951b71ed735be1b0" gracePeriod=30
Sep 18 19:47:17.057219 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:17.056648    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-bc6xr"
Sep 18 19:47:17.582245 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:17.582218    2564 generic.go:356] "Generic (PLEG): container finished" podID="e1919358-3891-4c29-a6ce-cfcde9675c53" containerID="af0dab12d92e1f7f308f1d067a465603d8019a40a2f22dcd951b71ed735be1b0" exitCode=0
Sep 18 19:47:17.582376 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:17.582290    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-bc6xr" event={"ID":"e1919358-3891-4c29-a6ce-cfcde9675c53","Type":"ContainerDied","Data":"af0dab12d92e1f7f308f1d067a465603d8019a40a2f22dcd951b71ed735be1b0"}
Sep 18 19:47:17.598136 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:17.598109    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-bc6xr"
Sep 18 19:47:17.708531 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:17.708493    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g887t\" (UniqueName: \"kubernetes.io/projected/e1919358-3891-4c29-a6ce-cfcde9675c53-kube-api-access-g887t\") pod \"e1919358-3891-4c29-a6ce-cfcde9675c53\" (UID: \"e1919358-3891-4c29-a6ce-cfcde9675c53\") "
Sep 18 19:47:17.708720 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:17.708571    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e1919358-3891-4c29-a6ce-cfcde9675c53-config-file\") pod \"e1919358-3891-4c29-a6ce-cfcde9675c53\" (UID: \"e1919358-3891-4c29-a6ce-cfcde9675c53\") "
Sep 18 19:47:17.708931 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:17.708904    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1919358-3891-4c29-a6ce-cfcde9675c53-config-file" (OuterVolumeSpecName: "config-file") pod "e1919358-3891-4c29-a6ce-cfcde9675c53" (UID: "e1919358-3891-4c29-a6ce-cfcde9675c53"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:47:17.710759 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:17.710731    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1919358-3891-4c29-a6ce-cfcde9675c53-kube-api-access-g887t" (OuterVolumeSpecName: "kube-api-access-g887t") pod "e1919358-3891-4c29-a6ce-cfcde9675c53" (UID: "e1919358-3891-4c29-a6ce-cfcde9675c53"). InnerVolumeSpecName "kube-api-access-g887t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:47:17.809318 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:17.809276    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g887t\" (UniqueName: \"kubernetes.io/projected/e1919358-3891-4c29-a6ce-cfcde9675c53-kube-api-access-g887t\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 19:47:17.809318 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:17.809309    2564 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e1919358-3891-4c29-a6ce-cfcde9675c53-config-file\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 19:47:18.587973 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:18.587940    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-bc6xr" event={"ID":"e1919358-3891-4c29-a6ce-cfcde9675c53","Type":"ContainerDied","Data":"4125bedba038321c9ddf2d03004fcd9c33b7686edb87490d65a094411b1652a3"}
Sep 18 19:47:18.588438 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:18.587951    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-bc6xr"
Sep 18 19:47:18.588438 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:18.588030    2564 scope.go:117] "RemoveContainer" containerID="af0dab12d92e1f7f308f1d067a465603d8019a40a2f22dcd951b71ed735be1b0"
Sep 18 19:47:18.602357 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:18.602324    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-bc6xr"]
Sep 18 19:47:18.606281 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:18.606250    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-bc6xr"]
Sep 18 19:47:19.999552 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:19.999516    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1919358-3891-4c29-a6ce-cfcde9675c53" path="/var/lib/kubelet/pods/e1919358-3891-4c29-a6ce-cfcde9675c53/volumes"
Sep 18 19:47:23.028447 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:23.028410    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:47:23.410391 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:23.410353    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:47:30.760994 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:30.760944    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:47:30.867632 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:30.867600    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:47:44.139634 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:44.139595    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-k6rr9"]
Sep 18 19:47:44.140016 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:44.139950    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1919358-3891-4c29-a6ce-cfcde9675c53" containerName="limitador"
Sep 18 19:47:44.140016 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:44.139963    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1919358-3891-4c29-a6ce-cfcde9675c53" containerName="limitador"
Sep 18 19:47:44.140097 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:44.140049    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1919358-3891-4c29-a6ce-cfcde9675c53" containerName="limitador"
Sep 18 19:47:44.143913 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:44.143894    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-k6rr9"
Sep 18 19:47:44.153383 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:44.153356    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-k6rr9"]
Sep 18 19:47:44.232874 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:44.232835    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/05a22ef8-c4cb-4d4e-a375-4eadb4213fc2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-k6rr9\" (UID: \"05a22ef8-c4cb-4d4e-a375-4eadb4213fc2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-k6rr9"
Sep 18 19:47:44.233098 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:44.232890    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/05a22ef8-c4cb-4d4e-a375-4eadb4213fc2-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-k6rr9\" (UID: \"05a22ef8-c4cb-4d4e-a375-4eadb4213fc2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-k6rr9"
Sep 18 19:47:44.233098 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:44.232911    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f7fb\" (UniqueName: \"kubernetes.io/projected/05a22ef8-c4cb-4d4e-a375-4eadb4213fc2-kube-api-access-6f7fb\") pod \"istiod-openshift-gateway-5677ccb49b-k6rr9\" (UID: \"05a22ef8-c4cb-4d4e-a375-4eadb4213fc2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-k6rr9"
Sep 18 19:47:44.233098 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:44.232935    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/05a22ef8-c4cb-4d4e-a375-4eadb4213fc2-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-k6rr9\" (UID: \"05a22ef8-c4cb-4d4e-a375-4eadb4213fc2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-k6rr9"
Sep 18 19:47:44.233098 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:44.232976    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05a22ef8-c4cb-4d4e-a375-4eadb4213fc2-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-k6rr9\" (UID: \"05a22ef8-c4cb-4d4e-a375-4eadb4213fc2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-k6rr9"
Sep 18 19:47:44.233098 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:44.233088    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/05a22ef8-c4cb-4d4e-a375-4eadb4213fc2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-k6rr9\" (UID: \"05a22ef8-c4cb-4d4e-a375-4eadb4213fc2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-k6rr9"
Sep 18 19:47:44.233272 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:44.233110    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/05a22ef8-c4cb-4d4e-a375-4eadb4213fc2-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-k6rr9\" (UID: \"05a22ef8-c4cb-4d4e-a375-4eadb4213fc2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-k6rr9"
Sep 18 19:47:44.334318 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:44.334280    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/05a22ef8-c4cb-4d4e-a375-4eadb4213fc2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-k6rr9\" (UID: \"05a22ef8-c4cb-4d4e-a375-4eadb4213fc2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-k6rr9"
Sep 18 19:47:44.334318 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:44.334321    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/05a22ef8-c4cb-4d4e-a375-4eadb4213fc2-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-k6rr9\" (UID: \"05a22ef8-c4cb-4d4e-a375-4eadb4213fc2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-k6rr9"
Sep 18 19:47:44.334534 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:44.334383    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/05a22ef8-c4cb-4d4e-a375-4eadb4213fc2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-k6rr9\" (UID: \"05a22ef8-c4cb-4d4e-a375-4eadb4213fc2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-k6rr9"
Sep 18 19:47:44.334534 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:44.334430    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/05a22ef8-c4cb-4d4e-a375-4eadb4213fc2-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-k6rr9\" (UID: \"05a22ef8-c4cb-4d4e-a375-4eadb4213fc2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-k6rr9"
Sep 18 19:47:44.334534 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:44.334453    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f7fb\" (UniqueName: \"kubernetes.io/projected/05a22ef8-c4cb-4d4e-a375-4eadb4213fc2-kube-api-access-6f7fb\") pod \"istiod-openshift-gateway-5677ccb49b-k6rr9\" (UID: \"05a22ef8-c4cb-4d4e-a375-4eadb4213fc2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-k6rr9"
Sep 18 19:47:44.334534 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:44.334488    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/05a22ef8-c4cb-4d4e-a375-4eadb4213fc2-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-k6rr9\" (UID: \"05a22ef8-c4cb-4d4e-a375-4eadb4213fc2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-k6rr9"
Sep 18 19:47:44.334534 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:44.334509    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05a22ef8-c4cb-4d4e-a375-4eadb4213fc2-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-k6rr9\" (UID: \"05a22ef8-c4cb-4d4e-a375-4eadb4213fc2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-k6rr9"
Sep 18 19:47:44.335273 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:44.335247    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/05a22ef8-c4cb-4d4e-a375-4eadb4213fc2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-k6rr9\" (UID: \"05a22ef8-c4cb-4d4e-a375-4eadb4213fc2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-k6rr9"
Sep 18 19:47:44.336824 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:44.336795    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/05a22ef8-c4cb-4d4e-a375-4eadb4213fc2-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-k6rr9\" (UID: \"05a22ef8-c4cb-4d4e-a375-4eadb4213fc2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-k6rr9"
Sep 18 19:47:44.336960 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:44.336940    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/05a22ef8-c4cb-4d4e-a375-4eadb4213fc2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-k6rr9\" (UID: \"05a22ef8-c4cb-4d4e-a375-4eadb4213fc2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-k6rr9"
Sep 18 19:47:44.337332 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:44.337310    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/05a22ef8-c4cb-4d4e-a375-4eadb4213fc2-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-k6rr9\" (UID: \"05a22ef8-c4cb-4d4e-a375-4eadb4213fc2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-k6rr9"
Sep 18 19:47:44.337403 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:44.337333    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/05a22ef8-c4cb-4d4e-a375-4eadb4213fc2-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-k6rr9\" (UID: \"05a22ef8-c4cb-4d4e-a375-4eadb4213fc2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-k6rr9"
Sep 18 19:47:44.343482 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:44.343452    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05a22ef8-c4cb-4d4e-a375-4eadb4213fc2-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-k6rr9\" (UID: \"05a22ef8-c4cb-4d4e-a375-4eadb4213fc2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-k6rr9"
Sep 18 19:47:44.343680 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:44.343662    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f7fb\" (UniqueName: \"kubernetes.io/projected/05a22ef8-c4cb-4d4e-a375-4eadb4213fc2-kube-api-access-6f7fb\") pod \"istiod-openshift-gateway-5677ccb49b-k6rr9\" (UID: \"05a22ef8-c4cb-4d4e-a375-4eadb4213fc2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-k6rr9"
Sep 18 19:47:44.454316 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:44.454216    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-k6rr9"
Sep 18 19:47:44.743968 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:44.743890    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-k6rr9"]
Sep 18 19:47:44.747357 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:47:44.747331    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05a22ef8_c4cb_4d4e_a375_4eadb4213fc2.slice/crio-ce5140799936a83388ebc932e2eaacd21ad2eec4e86545ac659c95d7b3689eb6 WatchSource:0}: Error finding container ce5140799936a83388ebc932e2eaacd21ad2eec4e86545ac659c95d7b3689eb6: Status 404 returned error can't find the container with id ce5140799936a83388ebc932e2eaacd21ad2eec4e86545ac659c95d7b3689eb6
Sep 18 19:47:44.749504 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:44.749472    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 18 19:47:44.749592 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:44.749537    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 18 19:47:45.728320 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:45.728290    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-k6rr9" event={"ID":"05a22ef8-c4cb-4d4e-a375-4eadb4213fc2","Type":"ContainerStarted","Data":"4664ab05f935adaf42bb0692d02a60d51983ac29308384a51667b713e9a1c0b9"}
Sep 18 19:47:45.728707 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:45.728341    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-k6rr9" event={"ID":"05a22ef8-c4cb-4d4e-a375-4eadb4213fc2","Type":"ContainerStarted","Data":"ce5140799936a83388ebc932e2eaacd21ad2eec4e86545ac659c95d7b3689eb6"}
Sep 18 19:47:45.728707 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:45.728586    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-k6rr9"
Sep 18 19:47:45.730274 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:45.730252    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-k6rr9"
Sep 18 19:47:45.766553 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:45.766497    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-k6rr9" podStartSLOduration=1.766478056 podStartE2EDuration="1.766478056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:47:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:47:45.747148834 +0000 UTC m=+582.257504926" watchObservedRunningTime="2026-09-18 19:47:45.766478056 +0000 UTC m=+582.276834147"
Sep 18 19:47:45.793850 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:45.793814    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nkzvl"]
Sep 18 19:47:45.794628 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:45.794326    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nkzvl" podUID="cea31a9b-a004-469c-904e-e68f8914fcc8" containerName="discovery" containerID="cri-o://960e44928654b7cea64634ac2dc5fc78d063c86d5e8fcc3ad50a457e9379a223" gracePeriod=30
Sep 18 19:47:46.040756 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:46.040730    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nkzvl"
Sep 18 19:47:46.152479 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:46.152437    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cea31a9b-a004-469c-904e-e68f8914fcc8-istio-csr-ca-configmap\") pod \"cea31a9b-a004-469c-904e-e68f8914fcc8\" (UID: \"cea31a9b-a004-469c-904e-e68f8914fcc8\") "
Sep 18 19:47:46.152685 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:46.152499    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l5c2x\" (UniqueName: \"kubernetes.io/projected/cea31a9b-a004-469c-904e-e68f8914fcc8-kube-api-access-l5c2x\") pod \"cea31a9b-a004-469c-904e-e68f8914fcc8\" (UID: \"cea31a9b-a004-469c-904e-e68f8914fcc8\") "
Sep 18 19:47:46.152685 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:46.152519    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cea31a9b-a004-469c-904e-e68f8914fcc8-cacerts\") pod \"cea31a9b-a004-469c-904e-e68f8914fcc8\" (UID: \"cea31a9b-a004-469c-904e-e68f8914fcc8\") "
Sep 18 19:47:46.152685 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:46.152540    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cea31a9b-a004-469c-904e-e68f8914fcc8-local-certs\") pod \"cea31a9b-a004-469c-904e-e68f8914fcc8\" (UID: \"cea31a9b-a004-469c-904e-e68f8914fcc8\") "
Sep 18 19:47:46.152685 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:46.152558    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cea31a9b-a004-469c-904e-e68f8914fcc8-istio-csr-dns-cert\") pod \"cea31a9b-a004-469c-904e-e68f8914fcc8\" (UID: \"cea31a9b-a004-469c-904e-e68f8914fcc8\") "
Sep 18 19:47:46.152685 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:46.152588    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cea31a9b-a004-469c-904e-e68f8914fcc8-istio-kubeconfig\") pod \"cea31a9b-a004-469c-904e-e68f8914fcc8\" (UID: \"cea31a9b-a004-469c-904e-e68f8914fcc8\") "
Sep 18 19:47:46.152685 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:46.152674    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cea31a9b-a004-469c-904e-e68f8914fcc8-istio-token\") pod \"cea31a9b-a004-469c-904e-e68f8914fcc8\" (UID: \"cea31a9b-a004-469c-904e-e68f8914fcc8\") "
Sep 18 19:47:46.153121 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:46.153082    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cea31a9b-a004-469c-904e-e68f8914fcc8-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "cea31a9b-a004-469c-904e-e68f8914fcc8" (UID: "cea31a9b-a004-469c-904e-e68f8914fcc8"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:47:46.155181 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:46.155153    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cea31a9b-a004-469c-904e-e68f8914fcc8-cacerts" (OuterVolumeSpecName: "cacerts") pod "cea31a9b-a004-469c-904e-e68f8914fcc8" (UID: "cea31a9b-a004-469c-904e-e68f8914fcc8"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:47:46.155282 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:46.155219    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cea31a9b-a004-469c-904e-e68f8914fcc8-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "cea31a9b-a004-469c-904e-e68f8914fcc8" (UID: "cea31a9b-a004-469c-904e-e68f8914fcc8"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:47:46.155413 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:46.155380    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cea31a9b-a004-469c-904e-e68f8914fcc8-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "cea31a9b-a004-469c-904e-e68f8914fcc8" (UID: "cea31a9b-a004-469c-904e-e68f8914fcc8"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:47:46.155557 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:46.155536    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cea31a9b-a004-469c-904e-e68f8914fcc8-istio-token" (OuterVolumeSpecName: "istio-token") pod "cea31a9b-a004-469c-904e-e68f8914fcc8" (UID: "cea31a9b-a004-469c-904e-e68f8914fcc8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:47:46.155625 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:46.155610    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cea31a9b-a004-469c-904e-e68f8914fcc8-kube-api-access-l5c2x" (OuterVolumeSpecName: "kube-api-access-l5c2x") pod "cea31a9b-a004-469c-904e-e68f8914fcc8" (UID: "cea31a9b-a004-469c-904e-e68f8914fcc8"). InnerVolumeSpecName "kube-api-access-l5c2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:47:46.155666 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:46.155624    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cea31a9b-a004-469c-904e-e68f8914fcc8-local-certs" (OuterVolumeSpecName: "local-certs") pod "cea31a9b-a004-469c-904e-e68f8914fcc8" (UID: "cea31a9b-a004-469c-904e-e68f8914fcc8"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:47:46.254388 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:46.254345    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5c2x\" (UniqueName: \"kubernetes.io/projected/cea31a9b-a004-469c-904e-e68f8914fcc8-kube-api-access-l5c2x\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 19:47:46.254563 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:46.254397    2564 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cea31a9b-a004-469c-904e-e68f8914fcc8-cacerts\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 19:47:46.254563 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:46.254411    2564 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cea31a9b-a004-469c-904e-e68f8914fcc8-local-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 19:47:46.254563 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:46.254419    2564 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cea31a9b-a004-469c-904e-e68f8914fcc8-istio-csr-dns-cert\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 19:47:46.254563 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:46.254427    2564 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cea31a9b-a004-469c-904e-e68f8914fcc8-istio-kubeconfig\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 19:47:46.254563 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:46.254435    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cea31a9b-a004-469c-904e-e68f8914fcc8-istio-token\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 19:47:46.254563 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:46.254443    2564 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cea31a9b-a004-469c-904e-e68f8914fcc8-istio-csr-ca-configmap\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 19:47:46.735330 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:46.735299    2564 generic.go:356] "Generic (PLEG): container finished" podID="cea31a9b-a004-469c-904e-e68f8914fcc8" containerID="960e44928654b7cea64634ac2dc5fc78d063c86d5e8fcc3ad50a457e9379a223" exitCode=0
Sep 18 19:47:46.735753 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:46.735346    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nkzvl"
Sep 18 19:47:46.735753 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:46.735384    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nkzvl" event={"ID":"cea31a9b-a004-469c-904e-e68f8914fcc8","Type":"ContainerDied","Data":"960e44928654b7cea64634ac2dc5fc78d063c86d5e8fcc3ad50a457e9379a223"}
Sep 18 19:47:46.735753 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:46.735433    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nkzvl" event={"ID":"cea31a9b-a004-469c-904e-e68f8914fcc8","Type":"ContainerDied","Data":"6f8698d384d88831d3044ef339a33c140c846f3b1ba8bbe18ee0383874837cf9"}
Sep 18 19:47:46.735753 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:46.735454    2564 scope.go:117] "RemoveContainer" containerID="960e44928654b7cea64634ac2dc5fc78d063c86d5e8fcc3ad50a457e9379a223"
Sep 18 19:47:46.744665 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:46.744649    2564 scope.go:117] "RemoveContainer" containerID="960e44928654b7cea64634ac2dc5fc78d063c86d5e8fcc3ad50a457e9379a223"
Sep 18 19:47:46.744962 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:47:46.744941    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"960e44928654b7cea64634ac2dc5fc78d063c86d5e8fcc3ad50a457e9379a223\": container with ID starting with 960e44928654b7cea64634ac2dc5fc78d063c86d5e8fcc3ad50a457e9379a223 not found: ID does not exist" containerID="960e44928654b7cea64634ac2dc5fc78d063c86d5e8fcc3ad50a457e9379a223"
Sep 18 19:47:46.745037 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:46.744973    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"960e44928654b7cea64634ac2dc5fc78d063c86d5e8fcc3ad50a457e9379a223"} err="failed to get container status \"960e44928654b7cea64634ac2dc5fc78d063c86d5e8fcc3ad50a457e9379a223\": rpc error: code = NotFound desc = could not find container \"960e44928654b7cea64634ac2dc5fc78d063c86d5e8fcc3ad50a457e9379a223\": container with ID starting with 960e44928654b7cea64634ac2dc5fc78d063c86d5e8fcc3ad50a457e9379a223 not found: ID does not exist"
Sep 18 19:47:46.756703 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:46.756670    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nkzvl"]
Sep 18 19:47:46.760756 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:46.760725    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nkzvl"]
Sep 18 19:47:47.999430 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:47.999395    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cea31a9b-a004-469c-904e-e68f8914fcc8" path="/var/lib/kubelet/pods/cea31a9b-a004-469c-904e-e68f8914fcc8/volumes"
Sep 18 19:47:55.148317 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:55.148275    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:47:55.207150 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:55.207114    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-b9b5d54f8-zvx8f"]
Sep 18 19:47:55.207489 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:55.207474    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cea31a9b-a004-469c-904e-e68f8914fcc8" containerName="discovery"
Sep 18 19:47:55.207489 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:55.207489    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="cea31a9b-a004-469c-904e-e68f8914fcc8" containerName="discovery"
Sep 18 19:47:55.207587 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:55.207571    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="cea31a9b-a004-469c-904e-e68f8914fcc8" containerName="discovery"
Sep 18 19:47:55.212151 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:55.212124    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-b9b5d54f8-zvx8f"
Sep 18 19:47:55.214247 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:55.214217    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Sep 18 19:47:55.214383 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:55.214301    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-bqmgj\""
Sep 18 19:47:55.214755 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:55.214737    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 18 19:47:55.214837 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:55.214755    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 18 19:47:55.218027 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:55.218005    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-b9b5d54f8-zvx8f"]
Sep 18 19:47:55.226291 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:55.226265    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-76bdf4b755-9n2gc"]
Sep 18 19:47:55.229938 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:55.229899    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:47:55.230483 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:55.230463    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-76bdf4b755-9n2gc"
Sep 18 19:47:55.232967 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:55.232944    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 18 19:47:55.233100 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:55.232950    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-xdw8h\""
Sep 18 19:47:55.238236 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:55.238208    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-76bdf4b755-9n2gc"]
Sep 18 19:47:55.312839 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:55.312804    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:47:55.333041 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:55.332966    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2fs8\" (UniqueName: \"kubernetes.io/projected/aa507a8c-3237-4c2c-8ce7-82dd6ca706b2-kube-api-access-x2fs8\") pod \"llmisvc-controller-manager-76bdf4b755-9n2gc\" (UID: \"aa507a8c-3237-4c2c-8ce7-82dd6ca706b2\") " pod="kserve/llmisvc-controller-manager-76bdf4b755-9n2gc"
Sep 18 19:47:55.333187 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:55.333047    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs4kc\" (UniqueName: \"kubernetes.io/projected/82041dd6-b911-4dc0-ba67-f612646666e9-kube-api-access-xs4kc\") pod \"kserve-controller-manager-b9b5d54f8-zvx8f\" (UID: \"82041dd6-b911-4dc0-ba67-f612646666e9\") " pod="kserve/kserve-controller-manager-b9b5d54f8-zvx8f"
Sep 18 19:47:55.333187 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:55.333166    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/82041dd6-b911-4dc0-ba67-f612646666e9-cert\") pod \"kserve-controller-manager-b9b5d54f8-zvx8f\" (UID: \"82041dd6-b911-4dc0-ba67-f612646666e9\") " pod="kserve/kserve-controller-manager-b9b5d54f8-zvx8f"
Sep 18 19:47:55.333261 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:55.333215    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aa507a8c-3237-4c2c-8ce7-82dd6ca706b2-cert\") pod \"llmisvc-controller-manager-76bdf4b755-9n2gc\" (UID: \"aa507a8c-3237-4c2c-8ce7-82dd6ca706b2\") " pod="kserve/llmisvc-controller-manager-76bdf4b755-9n2gc"
Sep 18 19:47:55.429744 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:55.429653    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:47:55.433661 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:55.433636    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/82041dd6-b911-4dc0-ba67-f612646666e9-cert\") pod \"kserve-controller-manager-b9b5d54f8-zvx8f\" (UID: \"82041dd6-b911-4dc0-ba67-f612646666e9\") " pod="kserve/kserve-controller-manager-b9b5d54f8-zvx8f"
Sep 18 19:47:55.433759 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:55.433677    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aa507a8c-3237-4c2c-8ce7-82dd6ca706b2-cert\") pod \"llmisvc-controller-manager-76bdf4b755-9n2gc\" (UID: \"aa507a8c-3237-4c2c-8ce7-82dd6ca706b2\") " pod="kserve/llmisvc-controller-manager-76bdf4b755-9n2gc"
Sep 18 19:47:55.433821 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:47:55.433771    2564 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 18 19:47:55.433821 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:55.433806    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2fs8\" (UniqueName: \"kubernetes.io/projected/aa507a8c-3237-4c2c-8ce7-82dd6ca706b2-kube-api-access-x2fs8\") pod \"llmisvc-controller-manager-76bdf4b755-9n2gc\" (UID: \"aa507a8c-3237-4c2c-8ce7-82dd6ca706b2\") " pod="kserve/llmisvc-controller-manager-76bdf4b755-9n2gc"
Sep 18 19:47:55.433919 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:47:55.433832    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82041dd6-b911-4dc0-ba67-f612646666e9-cert podName:82041dd6-b911-4dc0-ba67-f612646666e9 nodeName:}" failed. No retries permitted until 2026-09-18 19:47:55.933817375 +0000 UTC m=+592.444173443 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/82041dd6-b911-4dc0-ba67-f612646666e9-cert") pod "kserve-controller-manager-b9b5d54f8-zvx8f" (UID: "82041dd6-b911-4dc0-ba67-f612646666e9") : secret "kserve-webhook-server-cert" not found
Sep 18 19:47:55.433919 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:55.433857    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs4kc\" (UniqueName: \"kubernetes.io/projected/82041dd6-b911-4dc0-ba67-f612646666e9-kube-api-access-xs4kc\") pod \"kserve-controller-manager-b9b5d54f8-zvx8f\" (UID: \"82041dd6-b911-4dc0-ba67-f612646666e9\") " pod="kserve/kserve-controller-manager-b9b5d54f8-zvx8f"
Sep 18 19:47:55.436135 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:55.436117    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aa507a8c-3237-4c2c-8ce7-82dd6ca706b2-cert\") pod \"llmisvc-controller-manager-76bdf4b755-9n2gc\" (UID: \"aa507a8c-3237-4c2c-8ce7-82dd6ca706b2\") " pod="kserve/llmisvc-controller-manager-76bdf4b755-9n2gc"
Sep 18 19:47:55.444101 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:55.444064    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs4kc\" (UniqueName: \"kubernetes.io/projected/82041dd6-b911-4dc0-ba67-f612646666e9-kube-api-access-xs4kc\") pod \"kserve-controller-manager-b9b5d54f8-zvx8f\" (UID: \"82041dd6-b911-4dc0-ba67-f612646666e9\") " pod="kserve/kserve-controller-manager-b9b5d54f8-zvx8f"
Sep 18 19:47:55.444442 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:55.444425    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2fs8\" (UniqueName: \"kubernetes.io/projected/aa507a8c-3237-4c2c-8ce7-82dd6ca706b2-kube-api-access-x2fs8\") pod \"llmisvc-controller-manager-76bdf4b755-9n2gc\" (UID: \"aa507a8c-3237-4c2c-8ce7-82dd6ca706b2\") " pod="kserve/llmisvc-controller-manager-76bdf4b755-9n2gc"
Sep 18 19:47:55.529894 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:55.529858    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:47:55.542473 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:55.542445    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-76bdf4b755-9n2gc"
Sep 18 19:47:55.713549 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:55.713457    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:47:55.808627 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:55.808584    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:47:55.816074 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:55.816045    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-76bdf4b755-9n2gc"]
Sep 18 19:47:55.818680 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:47:55.818655    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podaa507a8c_3237_4c2c_8ce7_82dd6ca706b2.slice/crio-41a775f1f713010495c6825a87dc48b65ff80f691b77e03a5df56d3c51920047 WatchSource:0}: Error finding container 41a775f1f713010495c6825a87dc48b65ff80f691b77e03a5df56d3c51920047: Status 404 returned error can't find the container with id 41a775f1f713010495c6825a87dc48b65ff80f691b77e03a5df56d3c51920047
Sep 18 19:47:55.909197 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:55.909130    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:47:55.938328 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:55.938284    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/82041dd6-b911-4dc0-ba67-f612646666e9-cert\") pod \"kserve-controller-manager-b9b5d54f8-zvx8f\" (UID: \"82041dd6-b911-4dc0-ba67-f612646666e9\") " pod="kserve/kserve-controller-manager-b9b5d54f8-zvx8f"
Sep 18 19:47:55.940856 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:55.940833    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/82041dd6-b911-4dc0-ba67-f612646666e9-cert\") pod \"kserve-controller-manager-b9b5d54f8-zvx8f\" (UID: \"82041dd6-b911-4dc0-ba67-f612646666e9\") " pod="kserve/kserve-controller-manager-b9b5d54f8-zvx8f"
Sep 18 19:47:56.126119 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:56.125972    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-b9b5d54f8-zvx8f"
Sep 18 19:47:56.411595 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:56.411315    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-b9b5d54f8-zvx8f"]
Sep 18 19:47:56.414138 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:47:56.414107    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82041dd6_b911_4dc0_ba67_f612646666e9.slice/crio-76d5db3bbc4cd65186e809634229997e9a2c313a49355204cf95c274d949ecfd WatchSource:0}: Error finding container 76d5db3bbc4cd65186e809634229997e9a2c313a49355204cf95c274d949ecfd: Status 404 returned error can't find the container with id 76d5db3bbc4cd65186e809634229997e9a2c313a49355204cf95c274d949ecfd
Sep 18 19:47:56.791506 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:56.791401    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-b9b5d54f8-zvx8f" event={"ID":"82041dd6-b911-4dc0-ba67-f612646666e9","Type":"ContainerStarted","Data":"76d5db3bbc4cd65186e809634229997e9a2c313a49355204cf95c274d949ecfd"}
Sep 18 19:47:56.793248 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:56.793180    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-76bdf4b755-9n2gc" event={"ID":"aa507a8c-3237-4c2c-8ce7-82dd6ca706b2","Type":"ContainerStarted","Data":"41a775f1f713010495c6825a87dc48b65ff80f691b77e03a5df56d3c51920047"}
Sep 18 19:47:59.812647 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:59.812613    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-b9b5d54f8-zvx8f" event={"ID":"82041dd6-b911-4dc0-ba67-f612646666e9","Type":"ContainerStarted","Data":"923d2c2278b678ecbfb76920d85b0862761e1bbf32262c81fd676a5bd993d40d"}
Sep 18 19:47:59.813152 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:59.812702    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-b9b5d54f8-zvx8f"
Sep 18 19:47:59.814701 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:59.814674    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-76bdf4b755-9n2gc" event={"ID":"aa507a8c-3237-4c2c-8ce7-82dd6ca706b2","Type":"ContainerStarted","Data":"3884fccc10e5ead6145e7bb020553de2ce7420709a5003c7a5d25ff917a0dc57"}
Sep 18 19:47:59.814827 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:59.814726    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-76bdf4b755-9n2gc"
Sep 18 19:47:59.829460 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:59.829407    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-b9b5d54f8-zvx8f" podStartSLOduration=2.071822134 podStartE2EDuration="4.829383088s" totalImagesPullingTime="2.757560954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:47:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:47:59.826927646 +0000 UTC m=+596.337283761" watchObservedRunningTime="2026-09-18 19:47:59.829383088 +0000 UTC m=+596.339739178"
Sep 18 19:47:59.843473 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:47:59.843423    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-76bdf4b755-9n2gc" podStartSLOduration=1.493288947 podStartE2EDuration="4.843405395s" totalImagesPullingTime="3.350116448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:47:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:47:59.842353328 +0000 UTC m=+596.352709444" watchObservedRunningTime="2026-09-18 19:47:59.843405395 +0000 UTC m=+596.353761509"
Sep 18 19:48:03.912566 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:48:03.912537    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-58vl4_945a444e-9381-4e2f-b4c1-64be0ec88e6c/console-operator/2.log"
Sep 18 19:48:03.913028 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:48:03.912754    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-58vl4_945a444e-9381-4e2f-b4c1-64be0ec88e6c/console-operator/2.log"
Sep 18 19:48:30.822216 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:48:30.822179    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-76bdf4b755-9n2gc"
Sep 18 19:48:30.825271 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:48:30.825243    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-b9b5d54f8-zvx8f"
Sep 18 19:48:37.866538 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:48:37.866499    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-b9b5d54f8-zvx8f"]
Sep 18 19:48:37.866968 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:48:37.866699    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-b9b5d54f8-zvx8f" podUID="82041dd6-b911-4dc0-ba67-f612646666e9" containerName="manager" containerID="cri-o://923d2c2278b678ecbfb76920d85b0862761e1bbf32262c81fd676a5bd993d40d" gracePeriod=10
Sep 18 19:48:37.885786 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:48:37.885756    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-b9b5d54f8-vpxc9"]
Sep 18 19:48:37.889290 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:48:37.889271    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-b9b5d54f8-vpxc9"
Sep 18 19:48:37.896327 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:48:37.896300    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-b9b5d54f8-vpxc9"]
Sep 18 19:48:38.011998 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:48:38.011946    2564 generic.go:356] "Generic (PLEG): container finished" podID="82041dd6-b911-4dc0-ba67-f612646666e9" containerID="923d2c2278b678ecbfb76920d85b0862761e1bbf32262c81fd676a5bd993d40d" exitCode=0
Sep 18 19:48:38.012219 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:48:38.012027    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-b9b5d54f8-zvx8f" event={"ID":"82041dd6-b911-4dc0-ba67-f612646666e9","Type":"ContainerDied","Data":"923d2c2278b678ecbfb76920d85b0862761e1bbf32262c81fd676a5bd993d40d"}
Sep 18 19:48:38.026139 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:48:38.026100    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ccc8fe2a-105c-45ef-8f34-4e4b9a369e4d-cert\") pod \"kserve-controller-manager-b9b5d54f8-vpxc9\" (UID: \"ccc8fe2a-105c-45ef-8f34-4e4b9a369e4d\") " pod="kserve/kserve-controller-manager-b9b5d54f8-vpxc9"
Sep 18 19:48:38.026317 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:48:38.026224    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrtz8\" (UniqueName: \"kubernetes.io/projected/ccc8fe2a-105c-45ef-8f34-4e4b9a369e4d-kube-api-access-jrtz8\") pod \"kserve-controller-manager-b9b5d54f8-vpxc9\" (UID: \"ccc8fe2a-105c-45ef-8f34-4e4b9a369e4d\") " pod="kserve/kserve-controller-manager-b9b5d54f8-vpxc9"
Sep 18 19:48:38.124269 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:48:38.124204    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-b9b5d54f8-zvx8f"
Sep 18 19:48:38.127243 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:48:38.127210    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrtz8\" (UniqueName: \"kubernetes.io/projected/ccc8fe2a-105c-45ef-8f34-4e4b9a369e4d-kube-api-access-jrtz8\") pod \"kserve-controller-manager-b9b5d54f8-vpxc9\" (UID: \"ccc8fe2a-105c-45ef-8f34-4e4b9a369e4d\") " pod="kserve/kserve-controller-manager-b9b5d54f8-vpxc9"
Sep 18 19:48:38.127376 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:48:38.127260    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ccc8fe2a-105c-45ef-8f34-4e4b9a369e4d-cert\") pod \"kserve-controller-manager-b9b5d54f8-vpxc9\" (UID: \"ccc8fe2a-105c-45ef-8f34-4e4b9a369e4d\") " pod="kserve/kserve-controller-manager-b9b5d54f8-vpxc9"
Sep 18 19:48:38.129583 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:48:38.129558    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ccc8fe2a-105c-45ef-8f34-4e4b9a369e4d-cert\") pod \"kserve-controller-manager-b9b5d54f8-vpxc9\" (UID: \"ccc8fe2a-105c-45ef-8f34-4e4b9a369e4d\") " pod="kserve/kserve-controller-manager-b9b5d54f8-vpxc9"
Sep 18 19:48:38.134647 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:48:38.134620    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrtz8\" (UniqueName: \"kubernetes.io/projected/ccc8fe2a-105c-45ef-8f34-4e4b9a369e4d-kube-api-access-jrtz8\") pod \"kserve-controller-manager-b9b5d54f8-vpxc9\" (UID: \"ccc8fe2a-105c-45ef-8f34-4e4b9a369e4d\") " pod="kserve/kserve-controller-manager-b9b5d54f8-vpxc9"
Sep 18 19:48:38.227907 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:48:38.227862    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xs4kc\" (UniqueName: \"kubernetes.io/projected/82041dd6-b911-4dc0-ba67-f612646666e9-kube-api-access-xs4kc\") pod \"82041dd6-b911-4dc0-ba67-f612646666e9\" (UID: \"82041dd6-b911-4dc0-ba67-f612646666e9\") "
Sep 18 19:48:38.228094 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:48:38.227918    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/82041dd6-b911-4dc0-ba67-f612646666e9-cert\") pod \"82041dd6-b911-4dc0-ba67-f612646666e9\" (UID: \"82041dd6-b911-4dc0-ba67-f612646666e9\") "
Sep 18 19:48:38.230247 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:48:38.230214    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82041dd6-b911-4dc0-ba67-f612646666e9-kube-api-access-xs4kc" (OuterVolumeSpecName: "kube-api-access-xs4kc") pod "82041dd6-b911-4dc0-ba67-f612646666e9" (UID: "82041dd6-b911-4dc0-ba67-f612646666e9"). InnerVolumeSpecName "kube-api-access-xs4kc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:48:38.230365 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:48:38.230255    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82041dd6-b911-4dc0-ba67-f612646666e9-cert" (OuterVolumeSpecName: "cert") pod "82041dd6-b911-4dc0-ba67-f612646666e9" (UID: "82041dd6-b911-4dc0-ba67-f612646666e9"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:48:38.243473 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:48:38.243442    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-b9b5d54f8-vpxc9"
Sep 18 19:48:38.328964 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:48:38.328929    2564 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/82041dd6-b911-4dc0-ba67-f612646666e9-cert\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 19:48:38.328964 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:48:38.328962    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xs4kc\" (UniqueName: \"kubernetes.io/projected/82041dd6-b911-4dc0-ba67-f612646666e9-kube-api-access-xs4kc\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 19:48:38.516397 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:48:38.516364    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-b9b5d54f8-vpxc9"]
Sep 18 19:48:38.518395 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:48:38.518368    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccc8fe2a_105c_45ef_8f34_4e4b9a369e4d.slice/crio-adbf930a837f291fb415a081118b75f4e1cb262e51c34481d42f5cf7abb0a327 WatchSource:0}: Error finding container adbf930a837f291fb415a081118b75f4e1cb262e51c34481d42f5cf7abb0a327: Status 404 returned error can't find the container with id adbf930a837f291fb415a081118b75f4e1cb262e51c34481d42f5cf7abb0a327
Sep 18 19:48:39.019218 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:48:39.019185    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-b9b5d54f8-zvx8f"
Sep 18 19:48:39.019720 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:48:39.019182    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-b9b5d54f8-zvx8f" event={"ID":"82041dd6-b911-4dc0-ba67-f612646666e9","Type":"ContainerDied","Data":"76d5db3bbc4cd65186e809634229997e9a2c313a49355204cf95c274d949ecfd"}
Sep 18 19:48:39.019720 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:48:39.019334    2564 scope.go:117] "RemoveContainer" containerID="923d2c2278b678ecbfb76920d85b0862761e1bbf32262c81fd676a5bd993d40d"
Sep 18 19:48:39.021228 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:48:39.021186    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-b9b5d54f8-vpxc9" event={"ID":"ccc8fe2a-105c-45ef-8f34-4e4b9a369e4d","Type":"ContainerStarted","Data":"7bf563306f5aea627b2a4df8beaf846b1ec234e4f1718282490af8e459a408f1"}
Sep 18 19:48:39.021357 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:48:39.021246    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-b9b5d54f8-vpxc9" event={"ID":"ccc8fe2a-105c-45ef-8f34-4e4b9a369e4d","Type":"ContainerStarted","Data":"adbf930a837f291fb415a081118b75f4e1cb262e51c34481d42f5cf7abb0a327"}
Sep 18 19:48:39.021357 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:48:39.021281    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-b9b5d54f8-vpxc9"
Sep 18 19:48:39.036337 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:48:39.036286    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-b9b5d54f8-vpxc9" podStartSLOduration=1.619548593 podStartE2EDuration="2.036268696s" totalImagesPullingTime="416.720103ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:48:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:48:39.035239944 +0000 UTC m=+635.545596052" watchObservedRunningTime="2026-09-18 19:48:39.036268696 +0000 UTC m=+635.546624786"
Sep 18 19:48:39.049487 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:48:39.049456    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-b9b5d54f8-zvx8f"]
Sep 18 19:48:39.052709 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:48:39.052680    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-b9b5d54f8-zvx8f"]
Sep 18 19:48:39.999536 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:48:39.999506    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82041dd6-b911-4dc0-ba67-f612646666e9" path="/var/lib/kubelet/pods/82041dd6-b911-4dc0-ba67-f612646666e9/volumes"
Sep 18 19:49:10.031036 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:49:10.031001    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-b9b5d54f8-vpxc9"
Sep 18 19:49:11.251282 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:49:11.251242    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:49:11.292715 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:49:11.292682    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-xh6rg"]
Sep 18 19:49:11.293160 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:49:11.293141    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82041dd6-b911-4dc0-ba67-f612646666e9" containerName="manager"
Sep 18 19:49:11.293160 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:49:11.293162    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="82041dd6-b911-4dc0-ba67-f612646666e9" containerName="manager"
Sep 18 19:49:11.293330 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:49:11.293274    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="82041dd6-b911-4dc0-ba67-f612646666e9" containerName="manager"
Sep 18 19:49:11.297478 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:49:11.297452    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-xh6rg"
Sep 18 19:49:11.301207 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:49:11.301163    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-skrgh\""
Sep 18 19:49:11.301417 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:49:11.301397    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 18 19:49:11.301769 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:49:11.301728    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 18 19:49:11.311101 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:49:11.311070    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-xh6rg"]
Sep 18 19:49:11.413475 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:49:11.413268    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/881044d1-5a58-4322-8903-63bee4b44ddd-metrics-certs\") pod \"odh-model-controller-7b966d7984-xh6rg\" (UID: \"881044d1-5a58-4322-8903-63bee4b44ddd\") " pod="kserve/odh-model-controller-7b966d7984-xh6rg"
Sep 18 19:49:11.413475 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:49:11.413357    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/881044d1-5a58-4322-8903-63bee4b44ddd-cert\") pod \"odh-model-controller-7b966d7984-xh6rg\" (UID: \"881044d1-5a58-4322-8903-63bee4b44ddd\") " pod="kserve/odh-model-controller-7b966d7984-xh6rg"
Sep 18 19:49:11.413475 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:49:11.413396    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr86p\" (UniqueName: \"kubernetes.io/projected/881044d1-5a58-4322-8903-63bee4b44ddd-kube-api-access-nr86p\") pod \"odh-model-controller-7b966d7984-xh6rg\" (UID: \"881044d1-5a58-4322-8903-63bee4b44ddd\") " pod="kserve/odh-model-controller-7b966d7984-xh6rg"
Sep 18 19:49:11.514561 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:49:11.514448    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/881044d1-5a58-4322-8903-63bee4b44ddd-cert\") pod \"odh-model-controller-7b966d7984-xh6rg\" (UID: \"881044d1-5a58-4322-8903-63bee4b44ddd\") " pod="kserve/odh-model-controller-7b966d7984-xh6rg"
Sep 18 19:49:11.514561 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:49:11.514508    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr86p\" (UniqueName: \"kubernetes.io/projected/881044d1-5a58-4322-8903-63bee4b44ddd-kube-api-access-nr86p\") pod \"odh-model-controller-7b966d7984-xh6rg\" (UID: \"881044d1-5a58-4322-8903-63bee4b44ddd\") " pod="kserve/odh-model-controller-7b966d7984-xh6rg"
Sep 18 19:49:11.514803 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:49:11.514589    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/881044d1-5a58-4322-8903-63bee4b44ddd-metrics-certs\") pod \"odh-model-controller-7b966d7984-xh6rg\" (UID: \"881044d1-5a58-4322-8903-63bee4b44ddd\") " pod="kserve/odh-model-controller-7b966d7984-xh6rg"
Sep 18 19:49:11.514803 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:49:11.514629    2564 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 18 19:49:11.514803 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:49:11.514713    2564 secret.go:189] Couldn't get secret kserve/odh-model-controller-metrics-tls: secret "odh-model-controller-metrics-tls" not found
Sep 18 19:49:11.514803 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:49:11.514721    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/881044d1-5a58-4322-8903-63bee4b44ddd-cert podName:881044d1-5a58-4322-8903-63bee4b44ddd nodeName:}" failed. No retries permitted until 2026-09-18 19:49:12.014700749 +0000 UTC m=+668.525056820 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/881044d1-5a58-4322-8903-63bee4b44ddd-cert") pod "odh-model-controller-7b966d7984-xh6rg" (UID: "881044d1-5a58-4322-8903-63bee4b44ddd") : secret "odh-model-controller-webhook-cert" not found
Sep 18 19:49:11.514803 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:49:11.514777    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/881044d1-5a58-4322-8903-63bee4b44ddd-metrics-certs podName:881044d1-5a58-4322-8903-63bee4b44ddd nodeName:}" failed. No retries permitted until 2026-09-18 19:49:12.014758659 +0000 UTC m=+668.525114745 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/881044d1-5a58-4322-8903-63bee4b44ddd-metrics-certs") pod "odh-model-controller-7b966d7984-xh6rg" (UID: "881044d1-5a58-4322-8903-63bee4b44ddd") : secret "odh-model-controller-metrics-tls" not found
Sep 18 19:49:11.526230 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:49:11.526199    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr86p\" (UniqueName: \"kubernetes.io/projected/881044d1-5a58-4322-8903-63bee4b44ddd-kube-api-access-nr86p\") pod \"odh-model-controller-7b966d7984-xh6rg\" (UID: \"881044d1-5a58-4322-8903-63bee4b44ddd\") " pod="kserve/odh-model-controller-7b966d7984-xh6rg"
Sep 18 19:49:12.018733 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:49:12.018697    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/881044d1-5a58-4322-8903-63bee4b44ddd-cert\") pod \"odh-model-controller-7b966d7984-xh6rg\" (UID: \"881044d1-5a58-4322-8903-63bee4b44ddd\") " pod="kserve/odh-model-controller-7b966d7984-xh6rg"
Sep 18 19:49:12.018907 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:49:12.018766    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/881044d1-5a58-4322-8903-63bee4b44ddd-metrics-certs\") pod \"odh-model-controller-7b966d7984-xh6rg\" (UID: \"881044d1-5a58-4322-8903-63bee4b44ddd\") " pod="kserve/odh-model-controller-7b966d7984-xh6rg"
Sep 18 19:49:12.021230 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:49:12.021203    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/881044d1-5a58-4322-8903-63bee4b44ddd-metrics-certs\") pod \"odh-model-controller-7b966d7984-xh6rg\" (UID: \"881044d1-5a58-4322-8903-63bee4b44ddd\") " pod="kserve/odh-model-controller-7b966d7984-xh6rg"
Sep 18 19:49:12.021230 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:49:12.021215    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/881044d1-5a58-4322-8903-63bee4b44ddd-cert\") pod \"odh-model-controller-7b966d7984-xh6rg\" (UID: \"881044d1-5a58-4322-8903-63bee4b44ddd\") " pod="kserve/odh-model-controller-7b966d7984-xh6rg"
Sep 18 19:49:12.209736 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:49:12.209696    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-xh6rg"
Sep 18 19:49:12.465508 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:49:12.465469    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:49:12.481349 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:49:12.481324    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-xh6rg"]
Sep 18 19:49:12.483390 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:49:12.483360    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod881044d1_5a58_4322_8903_63bee4b44ddd.slice/crio-61f99ebd175b7ed65fb2968a4d106b9781449d6ce0ff3c363510b174bfb6f210 WatchSource:0}: Error finding container 61f99ebd175b7ed65fb2968a4d106b9781449d6ce0ff3c363510b174bfb6f210: Status 404 returned error can't find the container with id 61f99ebd175b7ed65fb2968a4d106b9781449d6ce0ff3c363510b174bfb6f210
Sep 18 19:49:12.484738 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:49:12.484719    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:49:13.181511 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:49:13.181471    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-xh6rg" event={"ID":"881044d1-5a58-4322-8903-63bee4b44ddd","Type":"ContainerStarted","Data":"61f99ebd175b7ed65fb2968a4d106b9781449d6ce0ff3c363510b174bfb6f210"}
Sep 18 19:49:15.904652 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:49:15.904616    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:49:16.198860 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:49:16.198763    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-xh6rg" event={"ID":"881044d1-5a58-4322-8903-63bee4b44ddd","Type":"ContainerStarted","Data":"284643ec195ecadad68f4a8683f26a21f809f7d002260171fe9b92954d7f4c67"}
Sep 18 19:49:16.199045 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:49:16.198865    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-xh6rg"
Sep 18 19:49:25.424170 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:49:25.424125    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:49:27.205789 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:49:27.205756    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-xh6rg"
Sep 18 19:49:27.221539 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:49:27.221474    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-xh6rg" podStartSLOduration=13.482225525 podStartE2EDuration="16.221456238s" totalImagesPullingTime="2.739230713s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:49:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:49:16.215010317 +0000 UTC m=+672.725366409" watchObservedRunningTime="2026-09-18 19:49:27.221456238 +0000 UTC m=+683.731812326"
Sep 18 19:50:05.555924 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:50:05.555830    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:50:05.616715 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:50:05.616680    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:50:31.925107 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:50:31.925061    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:50:42.462231 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:50:42.462183    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:50:42.514371 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:50:42.514342    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:50:45.223910 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:50:45.223871    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:50:45.324544 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:50:45.324488    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:50:57.297412 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:50:57.297351    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:50:57.394180 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:50:57.394143    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:51:12.553916 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:51:12.553877    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:51:21.197220 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:51:21.197172    2564 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-8699b59bbc-ggzph container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 18 19:51:21.197653 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:51:21.197256    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-ggzph" podUID="a1c0f546-0824-446c-8f25-ffed46184e1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 18 19:51:22.813246 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:51:22.813151    2564 generic.go:356] "Generic (PLEG): container finished" podID="a1c0f546-0824-446c-8f25-ffed46184e1c" containerID="8620bf11b83c057f761dfd87a965aefcf57d9958b869f2c6224ad25594c1b70b" exitCode=0
Sep 18 19:51:22.813246 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:51:22.813228    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-ggzph" event={"ID":"a1c0f546-0824-446c-8f25-ffed46184e1c","Type":"ContainerDied","Data":"8620bf11b83c057f761dfd87a965aefcf57d9958b869f2c6224ad25594c1b70b"}
Sep 18 19:51:22.814267 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:51:22.814235    2564 scope.go:117] "RemoveContainer" containerID="8620bf11b83c057f761dfd87a965aefcf57d9958b869f2c6224ad25594c1b70b"
Sep 18 19:51:22.815869 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:51:22.815824    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 18 19:51:22.816027 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:51:22.816001    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 18 19:51:22.816129 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:51:22.816105    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 18 19:51:23.821080 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:51:23.821026    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-ggzph" event={"ID":"a1c0f546-0824-446c-8f25-ffed46184e1c","Type":"ContainerStarted","Data":"c3d42d192f9302f307a5def3d3f5f00100e7a523e278fbd141dae6f71d2df412"}
Sep 18 19:51:23.900688 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:51:23.900641    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-ggzph"
Sep 18 19:51:23.900879 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:51:23.900863    2564 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-8699b59bbc-ggzph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Sep 18 19:51:23.900945 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:51:23.900924    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-ggzph" podUID="a1c0f546-0824-446c-8f25-ffed46184e1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Sep 18 19:51:24.900804 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:51:24.900764    2564 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-8699b59bbc-ggzph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Sep 18 19:51:24.901252 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:51:24.900837    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-ggzph" podUID="a1c0f546-0824-446c-8f25-ffed46184e1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Sep 18 19:51:25.900818 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:51:25.900782    2564 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-8699b59bbc-ggzph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Sep 18 19:51:25.901231 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:51:25.900844    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-ggzph" podUID="a1c0f546-0824-446c-8f25-ffed46184e1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Sep 18 19:51:26.900577 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:51:26.900536    2564 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-8699b59bbc-ggzph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Sep 18 19:51:26.900760 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:51:26.900597    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-ggzph" podUID="a1c0f546-0824-446c-8f25-ffed46184e1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Sep 18 19:51:27.900943 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:51:27.900898    2564 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-8699b59bbc-ggzph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Sep 18 19:51:27.901375 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:51:27.900963    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-ggzph" podUID="a1c0f546-0824-446c-8f25-ffed46184e1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Sep 18 19:51:28.900826 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:51:28.900778    2564 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-8699b59bbc-ggzph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Sep 18 19:51:28.901037 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:51:28.900847    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-ggzph" podUID="a1c0f546-0824-446c-8f25-ffed46184e1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Sep 18 19:51:29.904135 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:51:29.904103    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-ggzph"
Sep 18 19:51:29.904632 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:51:29.904611    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-ggzph"
Sep 18 19:51:29.905762 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:51:29.905732    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-ggzph"
Sep 18 19:51:30.532138 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:51:30.532105    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:51:30.799210 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:51:30.799122    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-59zrl"]
Sep 18 19:51:30.802099 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:51:30.802066    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-59zrl"
Sep 18 19:51:30.805242 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:51:30.805216    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Sep 18 19:51:30.805454 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:51:30.805216    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-epp-sa-dockercfg-fhp64\""
Sep 18 19:51:30.806026 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:51:30.805997    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Sep 18 19:51:30.806132 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:51:30.806045    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-8h2g5\""
Sep 18 19:51:30.806199 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:51:30.806127    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Sep 18 19:51:30.813218 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:51:30.813189    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-59zrl"]
Sep 18 19:51:30.839348 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:51:30.839311    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:51:30.886013 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:51:30.885964    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d8c4ce8-e4b3-47b5-be2d-281babe8a1a8-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-59zrl\" (UID: \"7d8c4ce8-e4b3-47b5-be2d-281babe8a1a8\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-59zrl"
Sep 18 19:51:30.886201 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:51:30.886078    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9l8d\" (UniqueName: \"kubernetes.io/projected/7d8c4ce8-e4b3-47b5-be2d-281babe8a1a8-kube-api-access-f9l8d\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-59zrl\" (UID: \"7d8c4ce8-e4b3-47b5-be2d-281babe8a1a8\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-59zrl"
Sep 18 19:51:30.987399 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:51:30.987356    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d8c4ce8-e4b3-47b5-be2d-281babe8a1a8-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-59zrl\" (UID: \"7d8c4ce8-e4b3-47b5-be2d-281babe8a1a8\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-59zrl"
Sep 18 19:51:30.987807 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:51:30.987475    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9l8d\" (UniqueName: \"kubernetes.io/projected/7d8c4ce8-e4b3-47b5-be2d-281babe8a1a8-kube-api-access-f9l8d\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-59zrl\" (UID: \"7d8c4ce8-e4b3-47b5-be2d-281babe8a1a8\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-59zrl"
Sep 18 19:51:30.990018 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:51:30.989961    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d8c4ce8-e4b3-47b5-be2d-281babe8a1a8-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-59zrl\" (UID: \"7d8c4ce8-e4b3-47b5-be2d-281babe8a1a8\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-59zrl"
Sep 18 19:51:30.995405 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:51:30.995375    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9l8d\" (UniqueName: \"kubernetes.io/projected/7d8c4ce8-e4b3-47b5-be2d-281babe8a1a8-kube-api-access-f9l8d\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-59zrl\" (UID: \"7d8c4ce8-e4b3-47b5-be2d-281babe8a1a8\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-59zrl"
Sep 18 19:51:31.117133 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:51:31.117024    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-59zrl"
Sep 18 19:51:31.393305 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:51:31.393269    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-59zrl"]
Sep 18 19:51:31.396542 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:51:31.396506    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d8c4ce8_e4b3_47b5_be2d_281babe8a1a8.slice/crio-beee997968339b35a3d2f7c1394eabea8004d6c9d70ae9f5e5f1715fd731bc7f WatchSource:0}: Error finding container beee997968339b35a3d2f7c1394eabea8004d6c9d70ae9f5e5f1715fd731bc7f: Status 404 returned error can't find the container with id beee997968339b35a3d2f7c1394eabea8004d6c9d70ae9f5e5f1715fd731bc7f
Sep 18 19:51:31.869970 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:51:31.869878    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-59zrl" event={"ID":"7d8c4ce8-e4b3-47b5-be2d-281babe8a1a8","Type":"ContainerStarted","Data":"beee997968339b35a3d2f7c1394eabea8004d6c9d70ae9f5e5f1715fd731bc7f"}
Sep 18 19:51:33.882292 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:51:33.882255    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-59zrl" event={"ID":"7d8c4ce8-e4b3-47b5-be2d-281babe8a1a8","Type":"ContainerStarted","Data":"a13c2025563ea33385c2e4f1fbb83704a195aa1b9bac74af45b6eb38cf463b98"}
Sep 18 19:51:33.882775 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:51:33.882535    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-59zrl"
Sep 18 19:51:33.898839 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:51:33.898778    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-59zrl" podStartSLOduration=1.543629201 podStartE2EDuration="3.898759559s" totalImagesPullingTime="2.355130358s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:51:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:51:33.897946086 +0000 UTC m=+810.408302202" watchObservedRunningTime="2026-09-18 19:51:33.898759559 +0000 UTC m=+810.409115651"
Sep 18 19:52:02.104727 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:52:02.104670    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:52:04.889728 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:52:04.889695    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-59zrl"
Sep 18 19:52:11.386043 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:52:11.386003    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-59zrl"]
Sep 18 19:52:11.394890 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:52:11.394832    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:52:11.447802 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:52:11.447765    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:52:14.890008 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:52:14.889944    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-59zrl" podUID="7d8c4ce8-e4b3-47b5-be2d-281babe8a1a8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:52:24.888952 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:52:24.888907    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-59zrl" podUID="7d8c4ce8-e4b3-47b5-be2d-281babe8a1a8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:52:26.408913 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:52:26.408841    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-59zrl" podUID="7d8c4ce8-e4b3-47b5-be2d-281babe8a1a8" containerName="main" containerID="cri-o://a13c2025563ea33385c2e4f1fbb83704a195aa1b9bac74af45b6eb38cf463b98" gracePeriod=45
Sep 18 19:52:28.677416 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:52:28.677375    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:52:28.779238 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:52:28.779191    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:52:34.889172 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:52:34.889130    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-59zrl" podUID="7d8c4ce8-e4b3-47b5-be2d-281babe8a1a8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:52:34.889695 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:52:34.889250    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-59zrl"
Sep 18 19:52:44.889231 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:52:44.889184    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-59zrl" podUID="7d8c4ce8-e4b3-47b5-be2d-281babe8a1a8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:52:54.889550 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:52:54.889504    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-59zrl" podUID="7d8c4ce8-e4b3-47b5-be2d-281babe8a1a8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:52:56.546776 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:52:56.546745    2564 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/systemd-tmpfiles-clean.service\": RecentStats: unable to find data in memory cache]"
Sep 18 19:52:56.669677 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:52:56.669608    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-59zrl"
Sep 18 19:52:56.755625 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:52:56.755580    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f9l8d\" (UniqueName: \"kubernetes.io/projected/7d8c4ce8-e4b3-47b5-be2d-281babe8a1a8-kube-api-access-f9l8d\") pod \"7d8c4ce8-e4b3-47b5-be2d-281babe8a1a8\" (UID: \"7d8c4ce8-e4b3-47b5-be2d-281babe8a1a8\") "
Sep 18 19:52:56.755806 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:52:56.755688    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d8c4ce8-e4b3-47b5-be2d-281babe8a1a8-tls-certs\") pod \"7d8c4ce8-e4b3-47b5-be2d-281babe8a1a8\" (UID: \"7d8c4ce8-e4b3-47b5-be2d-281babe8a1a8\") "
Sep 18 19:52:56.757849 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:52:56.757814    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d8c4ce8-e4b3-47b5-be2d-281babe8a1a8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7d8c4ce8-e4b3-47b5-be2d-281babe8a1a8" (UID: "7d8c4ce8-e4b3-47b5-be2d-281babe8a1a8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:52:56.757849 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:52:56.757839    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d8c4ce8-e4b3-47b5-be2d-281babe8a1a8-kube-api-access-f9l8d" (OuterVolumeSpecName: "kube-api-access-f9l8d") pod "7d8c4ce8-e4b3-47b5-be2d-281babe8a1a8" (UID: "7d8c4ce8-e4b3-47b5-be2d-281babe8a1a8"). InnerVolumeSpecName "kube-api-access-f9l8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:52:56.857016 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:52:56.856945    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d8c4ce8-e4b3-47b5-be2d-281babe8a1a8-tls-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 19:52:56.857016 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:52:56.857009    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9l8d\" (UniqueName: \"kubernetes.io/projected/7d8c4ce8-e4b3-47b5-be2d-281babe8a1a8-kube-api-access-f9l8d\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 19:52:57.290099 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:52:57.290065    2564 generic.go:356] "Generic (PLEG): container finished" podID="7d8c4ce8-e4b3-47b5-be2d-281babe8a1a8" containerID="a13c2025563ea33385c2e4f1fbb83704a195aa1b9bac74af45b6eb38cf463b98" exitCode=0
Sep 18 19:52:57.290303 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:52:57.290127    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-59zrl"
Sep 18 19:52:57.290303 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:52:57.290153    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-59zrl" event={"ID":"7d8c4ce8-e4b3-47b5-be2d-281babe8a1a8","Type":"ContainerDied","Data":"a13c2025563ea33385c2e4f1fbb83704a195aa1b9bac74af45b6eb38cf463b98"}
Sep 18 19:52:57.290303 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:52:57.290245    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-59zrl" event={"ID":"7d8c4ce8-e4b3-47b5-be2d-281babe8a1a8","Type":"ContainerDied","Data":"beee997968339b35a3d2f7c1394eabea8004d6c9d70ae9f5e5f1715fd731bc7f"}
Sep 18 19:52:57.290303 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:52:57.290276    2564 scope.go:117] "RemoveContainer" containerID="a13c2025563ea33385c2e4f1fbb83704a195aa1b9bac74af45b6eb38cf463b98"
Sep 18 19:52:57.299580 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:52:57.299555    2564 scope.go:117] "RemoveContainer" containerID="a13c2025563ea33385c2e4f1fbb83704a195aa1b9bac74af45b6eb38cf463b98"
Sep 18 19:52:57.299933 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:52:57.299913    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a13c2025563ea33385c2e4f1fbb83704a195aa1b9bac74af45b6eb38cf463b98\": container with ID starting with a13c2025563ea33385c2e4f1fbb83704a195aa1b9bac74af45b6eb38cf463b98 not found: ID does not exist" containerID="a13c2025563ea33385c2e4f1fbb83704a195aa1b9bac74af45b6eb38cf463b98"
Sep 18 19:52:57.300014 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:52:57.299945    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a13c2025563ea33385c2e4f1fbb83704a195aa1b9bac74af45b6eb38cf463b98"} err="failed to get container status \"a13c2025563ea33385c2e4f1fbb83704a195aa1b9bac74af45b6eb38cf463b98\": rpc error: code = NotFound desc = could not find container \"a13c2025563ea33385c2e4f1fbb83704a195aa1b9bac74af45b6eb38cf463b98\": container with ID starting with a13c2025563ea33385c2e4f1fbb83704a195aa1b9bac74af45b6eb38cf463b98 not found: ID does not exist"
Sep 18 19:52:57.310660 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:52:57.310627    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-59zrl"]
Sep 18 19:52:57.313922 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:52:57.313894    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-59zrl"]
Sep 18 19:52:58.000575 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:52:58.000538    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d8c4ce8-e4b3-47b5-be2d-281babe8a1a8" path="/var/lib/kubelet/pods/7d8c4ce8-e4b3-47b5-be2d-281babe8a1a8/volumes"
Sep 18 19:53:01.432940 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:53:01.432897    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:53:01.528080 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:53:01.528041    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:53:03.941411 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:53:03.941384    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-58vl4_945a444e-9381-4e2f-b4c1-64be0ec88e6c/console-operator/2.log"
Sep 18 19:53:03.942583 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:53:03.942551    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-58vl4_945a444e-9381-4e2f-b4c1-64be0ec88e6c/console-operator/2.log"
Sep 18 19:53:07.033965 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:53:07.033923    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:53:07.247656 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:53:07.247617    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-ncxg5"]
Sep 18 19:53:07.248219 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:53:07.248192    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d8c4ce8-e4b3-47b5-be2d-281babe8a1a8" containerName="main"
Sep 18 19:53:07.248405 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:53:07.248389    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d8c4ce8-e4b3-47b5-be2d-281babe8a1a8" containerName="main"
Sep 18 19:53:07.248659 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:53:07.248642    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d8c4ce8-e4b3-47b5-be2d-281babe8a1a8" containerName="main"
Sep 18 19:53:07.252082 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:53:07.252058    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-ncxg5"
Sep 18 19:53:07.255178 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:53:07.255154    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-epp-sa-dockercfg-js6ql\""
Sep 18 19:53:07.255395 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:53:07.255154    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-xm29g\""
Sep 18 19:53:07.255570 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:53:07.255154    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Sep 18 19:53:07.255642 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:53:07.255211    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Sep 18 19:53:07.255726 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:53:07.255238    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Sep 18 19:53:07.261237 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:53:07.261206    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-ncxg5"]
Sep 18 19:53:07.284337 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:53:07.284247    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:53:07.354395 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:53:07.354359    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0f72162-4732-41c0-9a3e-37133f0db397-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-ncxg5\" (UID: \"e0f72162-4732-41c0-9a3e-37133f0db397\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-ncxg5"
Sep 18 19:53:07.354588 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:53:07.354444    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9jgk\" (UniqueName: \"kubernetes.io/projected/e0f72162-4732-41c0-9a3e-37133f0db397-kube-api-access-h9jgk\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-ncxg5\" (UID: \"e0f72162-4732-41c0-9a3e-37133f0db397\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-ncxg5"
Sep 18 19:53:07.455818 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:53:07.455769    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9jgk\" (UniqueName: \"kubernetes.io/projected/e0f72162-4732-41c0-9a3e-37133f0db397-kube-api-access-h9jgk\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-ncxg5\" (UID: \"e0f72162-4732-41c0-9a3e-37133f0db397\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-ncxg5"
Sep 18 19:53:07.456059 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:53:07.455850    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0f72162-4732-41c0-9a3e-37133f0db397-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-ncxg5\" (UID: \"e0f72162-4732-41c0-9a3e-37133f0db397\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-ncxg5"
Sep 18 19:53:07.458378 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:53:07.458351    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0f72162-4732-41c0-9a3e-37133f0db397-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-ncxg5\" (UID: \"e0f72162-4732-41c0-9a3e-37133f0db397\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-ncxg5"
Sep 18 19:53:07.465429 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:53:07.465394    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9jgk\" (UniqueName: \"kubernetes.io/projected/e0f72162-4732-41c0-9a3e-37133f0db397-kube-api-access-h9jgk\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-ncxg5\" (UID: \"e0f72162-4732-41c0-9a3e-37133f0db397\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-ncxg5"
Sep 18 19:53:07.481094 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:53:07.481046    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:53:07.568130 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:53:07.568028    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-ncxg5"
Sep 18 19:53:07.839886 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:53:07.839862    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-ncxg5"]
Sep 18 19:53:07.842255 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:53:07.842212    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0f72162_4732_41c0_9a3e_37133f0db397.slice/crio-0b0fffb6cb57f8a5c94046dd9528733b60c64f61a720169aaa72ec528da926ff WatchSource:0}: Error finding container 0b0fffb6cb57f8a5c94046dd9528733b60c64f61a720169aaa72ec528da926ff: Status 404 returned error can't find the container with id 0b0fffb6cb57f8a5c94046dd9528733b60c64f61a720169aaa72ec528da926ff
Sep 18 19:53:08.346191 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:53:08.346154    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-ncxg5" event={"ID":"e0f72162-4732-41c0-9a3e-37133f0db397","Type":"ContainerStarted","Data":"294da04703a2331c0582b60856cee861b9aee97ec170ee7fef881f16a1b1eee5"}
Sep 18 19:53:08.346585 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:53:08.346219    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-ncxg5" event={"ID":"e0f72162-4732-41c0-9a3e-37133f0db397","Type":"ContainerStarted","Data":"0b0fffb6cb57f8a5c94046dd9528733b60c64f61a720169aaa72ec528da926ff"}
Sep 18 19:53:08.346585 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:53:08.346275    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-ncxg5"
Sep 18 19:53:08.363067 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:53:08.363015    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-ncxg5" podStartSLOduration=1.36299784 podStartE2EDuration="1.36299784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:53:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:53:08.361786196 +0000 UTC m=+904.872142314" watchObservedRunningTime="2026-09-18 19:53:08.36299784 +0000 UTC m=+904.873353923"
Sep 18 19:53:13.112422 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:53:13.112372    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:53:13.212453 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:53:13.212397    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:53:31.904673 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:53:31.904635    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:53:38.266790 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:53:38.266753    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:53:39.353075 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:53:39.353045    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-ncxg5"
Sep 18 19:53:45.957137 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:53:45.957096    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:53:45.957570 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:53:45.957265    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-ncxg5"]
Sep 18 19:53:46.047968 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:53:46.047925    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:53:49.352745 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:53:49.352703    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-ncxg5" podUID="e0f72162-4732-41c0-9a3e-37133f0db397" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:53:58.306881 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:53:58.306845    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:53:58.399504 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:53:58.399471    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:53:59.353360 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:53:59.353310    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-ncxg5" podUID="e0f72162-4732-41c0-9a3e-37133f0db397" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:54:00.978518 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:00.978467    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-ncxg5" podUID="e0f72162-4732-41c0-9a3e-37133f0db397" containerName="main" containerID="cri-o://294da04703a2331c0582b60856cee861b9aee97ec170ee7fef881f16a1b1eee5" gracePeriod=45
Sep 18 19:54:02.984635 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:02.984596    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:54:04.310771 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:04.310738    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:54:08.832300 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:08.832260    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:54:08.955792 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:08.955755    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:54:09.078120 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:09.078078    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:54:09.120340 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:09.120256    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche74hft"]
Sep 18 19:54:09.124074 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:09.124051    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche74hft"
Sep 18 19:54:09.127004 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:09.126952    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa-dockercfg-hms79\""
Sep 18 19:54:09.127149 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:09.127034    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 18 19:54:09.127149 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:09.127045    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 18 19:54:09.127149 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:09.126967    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 18 19:54:09.127149 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:09.127025    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-68qj6\""
Sep 18 19:54:09.135721 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:09.135695    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche74hft"]
Sep 18 19:54:09.186302 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:09.186273    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:54:09.229834 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:09.229798    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp8n5\" (UniqueName: \"kubernetes.io/projected/cd99db9e-1358-481a-812e-dcf9aed08100-kube-api-access-lp8n5\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche74hft\" (UID: \"cd99db9e-1358-481a-812e-dcf9aed08100\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche74hft"
Sep 18 19:54:09.230056 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:09.229853    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd99db9e-1358-481a-812e-dcf9aed08100-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche74hft\" (UID: \"cd99db9e-1358-481a-812e-dcf9aed08100\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche74hft"
Sep 18 19:54:09.331326 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:09.331290    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp8n5\" (UniqueName: \"kubernetes.io/projected/cd99db9e-1358-481a-812e-dcf9aed08100-kube-api-access-lp8n5\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche74hft\" (UID: \"cd99db9e-1358-481a-812e-dcf9aed08100\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche74hft"
Sep 18 19:54:09.331491 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:09.331336    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd99db9e-1358-481a-812e-dcf9aed08100-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche74hft\" (UID: \"cd99db9e-1358-481a-812e-dcf9aed08100\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche74hft"
Sep 18 19:54:09.333870 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:09.333841    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd99db9e-1358-481a-812e-dcf9aed08100-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche74hft\" (UID: \"cd99db9e-1358-481a-812e-dcf9aed08100\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche74hft"
Sep 18 19:54:09.353164 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:09.353118    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-ncxg5" podUID="e0f72162-4732-41c0-9a3e-37133f0db397" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:54:09.353318 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:09.353244    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-ncxg5"
Sep 18 19:54:09.356305 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:09.356278    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp8n5\" (UniqueName: \"kubernetes.io/projected/cd99db9e-1358-481a-812e-dcf9aed08100-kube-api-access-lp8n5\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche74hft\" (UID: \"cd99db9e-1358-481a-812e-dcf9aed08100\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche74hft"
Sep 18 19:54:09.437706 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:09.437659    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche74hft"
Sep 18 19:54:09.713541 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:09.713509    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche74hft"]
Sep 18 19:54:09.716174 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:54:09.716138    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd99db9e_1358_481a_812e_dcf9aed08100.slice/crio-68778343d6e1c5ee53b1a4e4ad3d88cb23c01d9933129d4f08fbac68bf1a1108 WatchSource:0}: Error finding container 68778343d6e1c5ee53b1a4e4ad3d88cb23c01d9933129d4f08fbac68bf1a1108: Status 404 returned error can't find the container with id 68778343d6e1c5ee53b1a4e4ad3d88cb23c01d9933129d4f08fbac68bf1a1108
Sep 18 19:54:10.633304 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:10.633260    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:54:10.654962 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:10.654927    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche74hft" event={"ID":"cd99db9e-1358-481a-812e-dcf9aed08100","Type":"ContainerStarted","Data":"8b19856456e0e9ad9516dabb262fba8bd02191aa7b0e8d02bd7c0219109bc650"}
Sep 18 19:54:10.655143 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:10.655017    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche74hft" event={"ID":"cd99db9e-1358-481a-812e-dcf9aed08100","Type":"ContainerStarted","Data":"68778343d6e1c5ee53b1a4e4ad3d88cb23c01d9933129d4f08fbac68bf1a1108"}
Sep 18 19:54:10.655143 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:10.655060    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche74hft"
Sep 18 19:54:10.670765 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:10.670702    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche74hft" podStartSLOduration=1.670680352 podStartE2EDuration="1.670680352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:54:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:54:10.669840572 +0000 UTC m=+967.180196690" watchObservedRunningTime="2026-09-18 19:54:10.670680352 +0000 UTC m=+967.181036443"
Sep 18 19:54:10.731507 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:10.731479    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:54:19.352930 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:19.352891    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-ncxg5" podUID="e0f72162-4732-41c0-9a3e-37133f0db397" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:54:28.011348 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:28.011244    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:54:29.352492 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:29.352443    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-ncxg5" podUID="e0f72162-4732-41c0-9a3e-37133f0db397" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:54:31.242592 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:31.242525    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-ncxg5"
Sep 18 19:54:31.317385 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:31.316678    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h9jgk\" (UniqueName: \"kubernetes.io/projected/e0f72162-4732-41c0-9a3e-37133f0db397-kube-api-access-h9jgk\") pod \"e0f72162-4732-41c0-9a3e-37133f0db397\" (UID: \"e0f72162-4732-41c0-9a3e-37133f0db397\") "
Sep 18 19:54:31.317385 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:31.316801    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0f72162-4732-41c0-9a3e-37133f0db397-tls-certs\") pod \"e0f72162-4732-41c0-9a3e-37133f0db397\" (UID: \"e0f72162-4732-41c0-9a3e-37133f0db397\") "
Sep 18 19:54:31.323077 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:31.323035    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0f72162-4732-41c0-9a3e-37133f0db397-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e0f72162-4732-41c0-9a3e-37133f0db397" (UID: "e0f72162-4732-41c0-9a3e-37133f0db397"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:54:31.327701 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:31.327359    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0f72162-4732-41c0-9a3e-37133f0db397-kube-api-access-h9jgk" (OuterVolumeSpecName: "kube-api-access-h9jgk") pod "e0f72162-4732-41c0-9a3e-37133f0db397" (UID: "e0f72162-4732-41c0-9a3e-37133f0db397"). InnerVolumeSpecName "kube-api-access-h9jgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:54:31.419008 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:31.418949    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9jgk\" (UniqueName: \"kubernetes.io/projected/e0f72162-4732-41c0-9a3e-37133f0db397-kube-api-access-h9jgk\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 19:54:31.419008 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:31.419001    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0f72162-4732-41c0-9a3e-37133f0db397-tls-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 19:54:31.758038 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:31.758006    2564 generic.go:356] "Generic (PLEG): container finished" podID="e0f72162-4732-41c0-9a3e-37133f0db397" containerID="294da04703a2331c0582b60856cee861b9aee97ec170ee7fef881f16a1b1eee5" exitCode=0
Sep 18 19:54:31.758226 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:31.758068    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-ncxg5"
Sep 18 19:54:31.758226 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:31.758098    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-ncxg5" event={"ID":"e0f72162-4732-41c0-9a3e-37133f0db397","Type":"ContainerDied","Data":"294da04703a2331c0582b60856cee861b9aee97ec170ee7fef881f16a1b1eee5"}
Sep 18 19:54:31.758226 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:31.758180    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-ncxg5" event={"ID":"e0f72162-4732-41c0-9a3e-37133f0db397","Type":"ContainerDied","Data":"0b0fffb6cb57f8a5c94046dd9528733b60c64f61a720169aaa72ec528da926ff"}
Sep 18 19:54:31.758226 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:31.758204    2564 scope.go:117] "RemoveContainer" containerID="294da04703a2331c0582b60856cee861b9aee97ec170ee7fef881f16a1b1eee5"
Sep 18 19:54:31.767400 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:31.767381    2564 scope.go:117] "RemoveContainer" containerID="294da04703a2331c0582b60856cee861b9aee97ec170ee7fef881f16a1b1eee5"
Sep 18 19:54:31.767709 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:54:31.767690    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"294da04703a2331c0582b60856cee861b9aee97ec170ee7fef881f16a1b1eee5\": container with ID starting with 294da04703a2331c0582b60856cee861b9aee97ec170ee7fef881f16a1b1eee5 not found: ID does not exist" containerID="294da04703a2331c0582b60856cee861b9aee97ec170ee7fef881f16a1b1eee5"
Sep 18 19:54:31.767761 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:31.767718    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"294da04703a2331c0582b60856cee861b9aee97ec170ee7fef881f16a1b1eee5"} err="failed to get container status \"294da04703a2331c0582b60856cee861b9aee97ec170ee7fef881f16a1b1eee5\": rpc error: code = NotFound desc = could not find container \"294da04703a2331c0582b60856cee861b9aee97ec170ee7fef881f16a1b1eee5\": container with ID starting with 294da04703a2331c0582b60856cee861b9aee97ec170ee7fef881f16a1b1eee5 not found: ID does not exist"
Sep 18 19:54:31.778272 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:31.778244    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-ncxg5"]
Sep 18 19:54:31.782063 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:31.782037    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-ncxg5"]
Sep 18 19:54:32.000613 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:32.000580    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0f72162-4732-41c0-9a3e-37133f0db397" path="/var/lib/kubelet/pods/e0f72162-4732-41c0-9a3e-37133f0db397/volumes"
Sep 18 19:54:35.977118 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:35.977084    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:54:36.076639 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:36.076603    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:54:36.202337 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:36.202304    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:54:36.322571 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:36.322460    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dqmt8"]
Sep 18 19:54:36.323100 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:36.323077    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0f72162-4732-41c0-9a3e-37133f0db397" containerName="main"
Sep 18 19:54:36.323100 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:36.323101    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0f72162-4732-41c0-9a3e-37133f0db397" containerName="main"
Sep 18 19:54:36.323303 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:36.323206    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0f72162-4732-41c0-9a3e-37133f0db397" containerName="main"
Sep 18 19:54:36.328055 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:36.328032    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dqmt8"
Sep 18 19:54:36.330588 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:36.330546    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"router-gateway-1-openshift-default-dockercfg-bjgbv\""
Sep 18 19:54:36.330588 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:36.330563    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-service-ca.crt\""
Sep 18 19:54:36.331168 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:36.331134    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"kube-root-ca.crt\""
Sep 18 19:54:36.331295 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:36.331183    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-gw-ca-root-cert\""
Sep 18 19:54:36.347171 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:36.343649    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dqmt8"]
Sep 18 19:54:36.362711 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:36.362649    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:54:36.363745 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:36.363721    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dbc067a-fed0-4e78-85fe-cfb811d36662-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5b4d9458bd-dqmt8\" (UID: \"5dbc067a-fed0-4e78-85fe-cfb811d36662\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dqmt8"
Sep 18 19:54:36.363886 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:36.363750    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dbc067a-fed0-4e78-85fe-cfb811d36662-istio-podinfo\") pod \"router-gateway-1-openshift-default-5b4d9458bd-dqmt8\" (UID: \"5dbc067a-fed0-4e78-85fe-cfb811d36662\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dqmt8"
Sep 18 19:54:36.363886 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:36.363770    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dbc067a-fed0-4e78-85fe-cfb811d36662-credential-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-dqmt8\" (UID: \"5dbc067a-fed0-4e78-85fe-cfb811d36662\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dqmt8"
Sep 18 19:54:36.364057 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:36.363886    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dbc067a-fed0-4e78-85fe-cfb811d36662-workload-certs\") pod \"router-gateway-1-openshift-default-5b4d9458bd-dqmt8\" (UID: \"5dbc067a-fed0-4e78-85fe-cfb811d36662\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dqmt8"
Sep 18 19:54:36.364057 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:36.363918    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dbc067a-fed0-4e78-85fe-cfb811d36662-istio-envoy\") pod \"router-gateway-1-openshift-default-5b4d9458bd-dqmt8\" (UID: \"5dbc067a-fed0-4e78-85fe-cfb811d36662\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dqmt8"
Sep 18 19:54:36.364057 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:36.363963    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dbc067a-fed0-4e78-85fe-cfb811d36662-workload-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-dqmt8\" (UID: \"5dbc067a-fed0-4e78-85fe-cfb811d36662\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dqmt8"
Sep 18 19:54:36.364057 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:36.364026    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99ctq\" (UniqueName: \"kubernetes.io/projected/5dbc067a-fed0-4e78-85fe-cfb811d36662-kube-api-access-99ctq\") pod \"router-gateway-1-openshift-default-5b4d9458bd-dqmt8\" (UID: \"5dbc067a-fed0-4e78-85fe-cfb811d36662\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dqmt8"
Sep 18 19:54:36.364057 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:36.364051    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dbc067a-fed0-4e78-85fe-cfb811d36662-istio-data\") pod \"router-gateway-1-openshift-default-5b4d9458bd-dqmt8\" (UID: \"5dbc067a-fed0-4e78-85fe-cfb811d36662\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dqmt8"
Sep 18 19:54:36.364343 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:36.364107    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dbc067a-fed0-4e78-85fe-cfb811d36662-istio-token\") pod \"router-gateway-1-openshift-default-5b4d9458bd-dqmt8\" (UID: \"5dbc067a-fed0-4e78-85fe-cfb811d36662\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dqmt8"
Sep 18 19:54:36.461019 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:36.460963    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:54:36.465017 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:36.464976    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dbc067a-fed0-4e78-85fe-cfb811d36662-workload-certs\") pod \"router-gateway-1-openshift-default-5b4d9458bd-dqmt8\" (UID: \"5dbc067a-fed0-4e78-85fe-cfb811d36662\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dqmt8"
Sep 18 19:54:36.465147 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:36.465030    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dbc067a-fed0-4e78-85fe-cfb811d36662-istio-envoy\") pod \"router-gateway-1-openshift-default-5b4d9458bd-dqmt8\" (UID: \"5dbc067a-fed0-4e78-85fe-cfb811d36662\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dqmt8"
Sep 18 19:54:36.465147 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:36.465067    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dbc067a-fed0-4e78-85fe-cfb811d36662-workload-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-dqmt8\" (UID: \"5dbc067a-fed0-4e78-85fe-cfb811d36662\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dqmt8"
Sep 18 19:54:36.465147 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:36.465092    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99ctq\" (UniqueName: \"kubernetes.io/projected/5dbc067a-fed0-4e78-85fe-cfb811d36662-kube-api-access-99ctq\") pod \"router-gateway-1-openshift-default-5b4d9458bd-dqmt8\" (UID: \"5dbc067a-fed0-4e78-85fe-cfb811d36662\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dqmt8"
Sep 18 19:54:36.465309 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:36.465235    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dbc067a-fed0-4e78-85fe-cfb811d36662-istio-data\") pod \"router-gateway-1-openshift-default-5b4d9458bd-dqmt8\" (UID: \"5dbc067a-fed0-4e78-85fe-cfb811d36662\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dqmt8"
Sep 18 19:54:36.465309 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:36.465296    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dbc067a-fed0-4e78-85fe-cfb811d36662-istio-token\") pod \"router-gateway-1-openshift-default-5b4d9458bd-dqmt8\" (UID: \"5dbc067a-fed0-4e78-85fe-cfb811d36662\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dqmt8"
Sep 18 19:54:36.465434 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:36.465339    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dbc067a-fed0-4e78-85fe-cfb811d36662-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5b4d9458bd-dqmt8\" (UID: \"5dbc067a-fed0-4e78-85fe-cfb811d36662\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dqmt8"
Sep 18 19:54:36.465434 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:36.465377    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dbc067a-fed0-4e78-85fe-cfb811d36662-istio-podinfo\") pod \"router-gateway-1-openshift-default-5b4d9458bd-dqmt8\" (UID: \"5dbc067a-fed0-4e78-85fe-cfb811d36662\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dqmt8"
Sep 18 19:54:36.465434 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:36.465410    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dbc067a-fed0-4e78-85fe-cfb811d36662-credential-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-dqmt8\" (UID: \"5dbc067a-fed0-4e78-85fe-cfb811d36662\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dqmt8"
Sep 18 19:54:36.465434 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:36.465409    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dbc067a-fed0-4e78-85fe-cfb811d36662-workload-certs\") pod \"router-gateway-1-openshift-default-5b4d9458bd-dqmt8\" (UID: \"5dbc067a-fed0-4e78-85fe-cfb811d36662\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dqmt8"
Sep 18 19:54:36.465630 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:36.465461    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dbc067a-fed0-4e78-85fe-cfb811d36662-workload-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-dqmt8\" (UID: \"5dbc067a-fed0-4e78-85fe-cfb811d36662\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dqmt8"
Sep 18 19:54:36.465806 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:36.465763    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dbc067a-fed0-4e78-85fe-cfb811d36662-credential-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-dqmt8\" (UID: \"5dbc067a-fed0-4e78-85fe-cfb811d36662\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dqmt8"
Sep 18 19:54:36.465941 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:36.465828    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dbc067a-fed0-4e78-85fe-cfb811d36662-istio-data\") pod \"router-gateway-1-openshift-default-5b4d9458bd-dqmt8\" (UID: \"5dbc067a-fed0-4e78-85fe-cfb811d36662\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dqmt8"
Sep 18 19:54:36.466071 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:36.466048    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dbc067a-fed0-4e78-85fe-cfb811d36662-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5b4d9458bd-dqmt8\" (UID: \"5dbc067a-fed0-4e78-85fe-cfb811d36662\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dqmt8"
Sep 18 19:54:36.467638 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:36.467610    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dbc067a-fed0-4e78-85fe-cfb811d36662-istio-envoy\") pod \"router-gateway-1-openshift-default-5b4d9458bd-dqmt8\" (UID: \"5dbc067a-fed0-4e78-85fe-cfb811d36662\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dqmt8"
Sep 18 19:54:36.467938 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:36.467917    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dbc067a-fed0-4e78-85fe-cfb811d36662-istio-podinfo\") pod \"router-gateway-1-openshift-default-5b4d9458bd-dqmt8\" (UID: \"5dbc067a-fed0-4e78-85fe-cfb811d36662\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dqmt8"
Sep 18 19:54:36.473115 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:36.473085    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dbc067a-fed0-4e78-85fe-cfb811d36662-istio-token\") pod \"router-gateway-1-openshift-default-5b4d9458bd-dqmt8\" (UID: \"5dbc067a-fed0-4e78-85fe-cfb811d36662\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dqmt8"
Sep 18 19:54:36.473222 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:36.473142    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99ctq\" (UniqueName: \"kubernetes.io/projected/5dbc067a-fed0-4e78-85fe-cfb811d36662-kube-api-access-99ctq\") pod \"router-gateway-1-openshift-default-5b4d9458bd-dqmt8\" (UID: \"5dbc067a-fed0-4e78-85fe-cfb811d36662\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dqmt8"
Sep 18 19:54:36.647074 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:36.647031    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dqmt8"
Sep 18 19:54:36.925895 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:36.925758    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dqmt8"]
Sep 18 19:54:36.928139 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:54:36.928111    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dbc067a_fed0_4e78_85fe_cfb811d36662.slice/crio-d4f39e03fb40011757c1e977a92c50973b2862f317b65aebc819d08628b72680 WatchSource:0}: Error finding container d4f39e03fb40011757c1e977a92c50973b2862f317b65aebc819d08628b72680: Status 404 returned error can't find the container with id d4f39e03fb40011757c1e977a92c50973b2862f317b65aebc819d08628b72680
Sep 18 19:54:36.930200 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:36.930168    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 18 19:54:36.930286 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:36.930236    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 18 19:54:36.930286 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:36.930270    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 18 19:54:37.792080 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:37.792047    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dqmt8" event={"ID":"5dbc067a-fed0-4e78-85fe-cfb811d36662","Type":"ContainerStarted","Data":"25fa8c53daf79bb2b1c6c09325ffdff5b5aadb11aef6d94c140e3597c468ecf0"}
Sep 18 19:54:37.792466 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:37.792099    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dqmt8" event={"ID":"5dbc067a-fed0-4e78-85fe-cfb811d36662","Type":"ContainerStarted","Data":"d4f39e03fb40011757c1e977a92c50973b2862f317b65aebc819d08628b72680"}
Sep 18 19:54:37.811127 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:37.811074    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dqmt8" podStartSLOduration=1.811055276 podStartE2EDuration="1.811055276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:54:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:54:37.808888991 +0000 UTC m=+994.319245081" watchObservedRunningTime="2026-09-18 19:54:37.811055276 +0000 UTC m=+994.321411368"
Sep 18 19:54:38.647578 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:38.647533    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dqmt8"
Sep 18 19:54:38.652975 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:38.652945    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dqmt8"
Sep 18 19:54:38.797457 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:38.797417    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dqmt8"
Sep 18 19:54:38.798373 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:38.798351    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dqmt8"
Sep 18 19:54:39.662813 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:39.662779    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:54:41.660717 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:41.660690    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche74hft"
Sep 18 19:54:42.239691 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:42.239651    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:54:47.701037 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:47.700999    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:54:47.759492 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:54:47.759430    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:55:17.113678 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:17.113636    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-sssbg"]
Sep 18 19:55:17.118667 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:17.118638    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-sssbg"
Sep 18 19:55:17.123689 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:17.123666    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"default-dockercfg-mv9d8\""
Sep 18 19:55:17.123838 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:17.123666    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs\""
Sep 18 19:55:17.126492 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:17.126469    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-sssbg"]
Sep 18 19:55:17.144596 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:17.144561    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b1a6fba-283e-4d58-8460-8ae01b6cb2a2-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-sssbg\" (UID: \"9b1a6fba-283e-4d58-8460-8ae01b6cb2a2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-sssbg"
Sep 18 19:55:17.144596 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:17.144599    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b1a6fba-283e-4d58-8460-8ae01b6cb2a2-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-sssbg\" (UID: \"9b1a6fba-283e-4d58-8460-8ae01b6cb2a2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-sssbg"
Sep 18 19:55:17.144847 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:17.144711    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b1a6fba-283e-4d58-8460-8ae01b6cb2a2-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-sssbg\" (UID: \"9b1a6fba-283e-4d58-8460-8ae01b6cb2a2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-sssbg"
Sep 18 19:55:17.144847 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:17.144740    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff78f\" (UniqueName: \"kubernetes.io/projected/9b1a6fba-283e-4d58-8460-8ae01b6cb2a2-kube-api-access-ff78f\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-sssbg\" (UID: \"9b1a6fba-283e-4d58-8460-8ae01b6cb2a2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-sssbg"
Sep 18 19:55:17.144847 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:17.144761    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b1a6fba-283e-4d58-8460-8ae01b6cb2a2-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-sssbg\" (UID: \"9b1a6fba-283e-4d58-8460-8ae01b6cb2a2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-sssbg"
Sep 18 19:55:17.144847 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:17.144833    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b1a6fba-283e-4d58-8460-8ae01b6cb2a2-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-sssbg\" (UID: \"9b1a6fba-283e-4d58-8460-8ae01b6cb2a2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-sssbg"
Sep 18 19:55:17.192540 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:17.192508    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:55:17.245535 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:17.245493    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b1a6fba-283e-4d58-8460-8ae01b6cb2a2-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-sssbg\" (UID: \"9b1a6fba-283e-4d58-8460-8ae01b6cb2a2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-sssbg"
Sep 18 19:55:17.246236 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:17.246181    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b1a6fba-283e-4d58-8460-8ae01b6cb2a2-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-sssbg\" (UID: \"9b1a6fba-283e-4d58-8460-8ae01b6cb2a2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-sssbg"
Sep 18 19:55:17.246534 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:17.246507    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b1a6fba-283e-4d58-8460-8ae01b6cb2a2-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-sssbg\" (UID: \"9b1a6fba-283e-4d58-8460-8ae01b6cb2a2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-sssbg"
Sep 18 19:55:17.246534 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:17.246530    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b1a6fba-283e-4d58-8460-8ae01b6cb2a2-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-sssbg\" (UID: \"9b1a6fba-283e-4d58-8460-8ae01b6cb2a2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-sssbg"
Sep 18 19:55:17.246714 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:17.246581    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff78f\" (UniqueName: \"kubernetes.io/projected/9b1a6fba-283e-4d58-8460-8ae01b6cb2a2-kube-api-access-ff78f\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-sssbg\" (UID: \"9b1a6fba-283e-4d58-8460-8ae01b6cb2a2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-sssbg"
Sep 18 19:55:17.246714 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:17.246612    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b1a6fba-283e-4d58-8460-8ae01b6cb2a2-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-sssbg\" (UID: \"9b1a6fba-283e-4d58-8460-8ae01b6cb2a2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-sssbg"
Sep 18 19:55:17.246714 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:17.246638    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b1a6fba-283e-4d58-8460-8ae01b6cb2a2-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-sssbg\" (UID: \"9b1a6fba-283e-4d58-8460-8ae01b6cb2a2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-sssbg"
Sep 18 19:55:17.246714 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:17.246663    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b1a6fba-283e-4d58-8460-8ae01b6cb2a2-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-sssbg\" (UID: \"9b1a6fba-283e-4d58-8460-8ae01b6cb2a2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-sssbg"
Sep 18 19:55:17.247151 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:17.247129    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b1a6fba-283e-4d58-8460-8ae01b6cb2a2-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-sssbg\" (UID: \"9b1a6fba-283e-4d58-8460-8ae01b6cb2a2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-sssbg"
Sep 18 19:55:17.249405 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:17.249378    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b1a6fba-283e-4d58-8460-8ae01b6cb2a2-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-sssbg\" (UID: \"9b1a6fba-283e-4d58-8460-8ae01b6cb2a2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-sssbg"
Sep 18 19:55:17.249684 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:17.249666    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b1a6fba-283e-4d58-8460-8ae01b6cb2a2-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-sssbg\" (UID: \"9b1a6fba-283e-4d58-8460-8ae01b6cb2a2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-sssbg"
Sep 18 19:55:17.254070 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:17.254050    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff78f\" (UniqueName: \"kubernetes.io/projected/9b1a6fba-283e-4d58-8460-8ae01b6cb2a2-kube-api-access-ff78f\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-sssbg\" (UID: \"9b1a6fba-283e-4d58-8460-8ae01b6cb2a2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-sssbg"
Sep 18 19:55:17.430191 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:17.430129    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-sssbg"
Sep 18 19:55:17.706768 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:17.706738    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-sssbg"]
Sep 18 19:55:17.709164 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:55:17.709130    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b1a6fba_283e_4d58_8460_8ae01b6cb2a2.slice/crio-cad248711f492eea175ce269cec6792e31a39efa990f735dbeb61cf0e9dd379d WatchSource:0}: Error finding container cad248711f492eea175ce269cec6792e31a39efa990f735dbeb61cf0e9dd379d: Status 404 returned error can't find the container with id cad248711f492eea175ce269cec6792e31a39efa990f735dbeb61cf0e9dd379d
Sep 18 19:55:17.711192 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:17.711172    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:55:17.986363 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:17.986280    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-sssbg" event={"ID":"9b1a6fba-283e-4d58-8460-8ae01b6cb2a2","Type":"ContainerStarted","Data":"cad248711f492eea175ce269cec6792e31a39efa990f735dbeb61cf0e9dd379d"}
Sep 18 19:55:19.424768 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:19.424697    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:55:19.526453 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:19.526368    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:55:19.626089 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:19.626039    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:55:19.769007 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:19.762143    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:55:19.891263 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:19.891218    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:55:20.001101 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:20.001008    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-sssbg" event={"ID":"9b1a6fba-283e-4d58-8460-8ae01b6cb2a2","Type":"ContainerStarted","Data":"7710cb84a6c6cec5413a184cfeda416a77ac77b704674c3323ae11966437e71a"}
Sep 18 19:55:20.021199 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:20.021148    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-sssbg" podStartSLOduration=0.99817643 podStartE2EDuration="3.021129172s" totalImagesPullingTime="2.022952742s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:55:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:55:20.019167917 +0000 UTC m=+1036.529524008" watchObservedRunningTime="2026-09-18 19:55:20.021129172 +0000 UTC m=+1036.531485263"
Sep 18 19:55:20.430943 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:20.430905    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-sssbg"
Sep 18 19:55:23.198574 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:23.198527    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:55:24.095177 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:24.095142    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-sssbg"]
Sep 18 19:55:24.104637 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:55:24.104588    2564 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 19:55:24.104637 ip-10-0-128-161 kubenswrapper[2564]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:55:24.104637 ip-10-0-128-161 kubenswrapper[2564]:         , stdout: , stderr: , exit code -1
Sep 18 19:55:24.104637 ip-10-0-128-161 kubenswrapper[2564]:  > containerID="7710cb84a6c6cec5413a184cfeda416a77ac77b704674c3323ae11966437e71a" cmd=["/bin/sleep","15"]
Sep 18 19:55:24.104886 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:24.104663    2564 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-sssbg" message="" err=<
Sep 18 19:55:24.104886 ip-10-0-128-161 kubenswrapper[2564]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:55:24.104886 ip-10-0-128-161 kubenswrapper[2564]:         , stdout: , stderr: , exit code -1
Sep 18 19:55:24.104886 ip-10-0-128-161 kubenswrapper[2564]:  >
Sep 18 19:55:24.104886 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:55:24.104687    2564 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 19:55:24.104886 ip-10-0-128-161 kubenswrapper[2564]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:55:24.104886 ip-10-0-128-161 kubenswrapper[2564]:         , stdout: , stderr: , exit code -1
Sep 18 19:55:24.104886 ip-10-0-128-161 kubenswrapper[2564]:  > pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-sssbg" podUID="9b1a6fba-283e-4d58-8460-8ae01b6cb2a2" containerName="main" containerID="cri-o://7710cb84a6c6cec5413a184cfeda416a77ac77b704674c3323ae11966437e71a"
Sep 18 19:55:24.104886 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:24.104738    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-sssbg" podUID="9b1a6fba-283e-4d58-8460-8ae01b6cb2a2" containerName="main" containerID="cri-o://7710cb84a6c6cec5413a184cfeda416a77ac77b704674c3323ae11966437e71a" gracePeriod=60
Sep 18 19:55:24.117569 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:24.117531    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:55:24.406193 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:24.406157    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dqmt8"]
Sep 18 19:55:24.406660 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:24.406389    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dqmt8" podUID="5dbc067a-fed0-4e78-85fe-cfb811d36662" containerName="istio-proxy" containerID="cri-o://25fa8c53daf79bb2b1c6c09325ffdff5b5aadb11aef6d94c140e3597c468ecf0" gracePeriod=30
Sep 18 19:55:24.449747 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:24.449703    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:55:24.463776 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:24.463700    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-sssbg"
Sep 18 19:55:24.525635 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:24.525601    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b1a6fba-283e-4d58-8460-8ae01b6cb2a2-model-cache\") pod \"9b1a6fba-283e-4d58-8460-8ae01b6cb2a2\" (UID: \"9b1a6fba-283e-4d58-8460-8ae01b6cb2a2\") "
Sep 18 19:55:24.525635 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:24.525637    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ff78f\" (UniqueName: \"kubernetes.io/projected/9b1a6fba-283e-4d58-8460-8ae01b6cb2a2-kube-api-access-ff78f\") pod \"9b1a6fba-283e-4d58-8460-8ae01b6cb2a2\" (UID: \"9b1a6fba-283e-4d58-8460-8ae01b6cb2a2\") "
Sep 18 19:55:24.525914 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:24.525709    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b1a6fba-283e-4d58-8460-8ae01b6cb2a2-home\") pod \"9b1a6fba-283e-4d58-8460-8ae01b6cb2a2\" (UID: \"9b1a6fba-283e-4d58-8460-8ae01b6cb2a2\") "
Sep 18 19:55:24.525914 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:24.525730    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b1a6fba-283e-4d58-8460-8ae01b6cb2a2-tls-certs\") pod \"9b1a6fba-283e-4d58-8460-8ae01b6cb2a2\" (UID: \"9b1a6fba-283e-4d58-8460-8ae01b6cb2a2\") "
Sep 18 19:55:24.525914 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:24.525748    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b1a6fba-283e-4d58-8460-8ae01b6cb2a2-dshm\") pod \"9b1a6fba-283e-4d58-8460-8ae01b6cb2a2\" (UID: \"9b1a6fba-283e-4d58-8460-8ae01b6cb2a2\") "
Sep 18 19:55:24.525914 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:24.525804    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b1a6fba-283e-4d58-8460-8ae01b6cb2a2-tmp-dir\") pod \"9b1a6fba-283e-4d58-8460-8ae01b6cb2a2\" (UID: \"9b1a6fba-283e-4d58-8460-8ae01b6cb2a2\") "
Sep 18 19:55:24.526154 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:24.525938    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b1a6fba-283e-4d58-8460-8ae01b6cb2a2-model-cache" (OuterVolumeSpecName: "model-cache") pod "9b1a6fba-283e-4d58-8460-8ae01b6cb2a2" (UID: "9b1a6fba-283e-4d58-8460-8ae01b6cb2a2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:55:24.526238 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:24.526203    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b1a6fba-283e-4d58-8460-8ae01b6cb2a2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9b1a6fba-283e-4d58-8460-8ae01b6cb2a2" (UID: "9b1a6fba-283e-4d58-8460-8ae01b6cb2a2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:55:24.526238 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:24.526216    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b1a6fba-283e-4d58-8460-8ae01b6cb2a2-home" (OuterVolumeSpecName: "home") pod "9b1a6fba-283e-4d58-8460-8ae01b6cb2a2" (UID: "9b1a6fba-283e-4d58-8460-8ae01b6cb2a2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:55:24.526421 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:24.526403    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b1a6fba-283e-4d58-8460-8ae01b6cb2a2-home\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 19:55:24.526482 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:24.526426    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b1a6fba-283e-4d58-8460-8ae01b6cb2a2-tmp-dir\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 19:55:24.526482 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:24.526440    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b1a6fba-283e-4d58-8460-8ae01b6cb2a2-model-cache\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 19:55:24.528494 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:24.528458    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b1a6fba-283e-4d58-8460-8ae01b6cb2a2-kube-api-access-ff78f" (OuterVolumeSpecName: "kube-api-access-ff78f") pod "9b1a6fba-283e-4d58-8460-8ae01b6cb2a2" (UID: "9b1a6fba-283e-4d58-8460-8ae01b6cb2a2"). InnerVolumeSpecName "kube-api-access-ff78f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:55:24.528494 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:24.528481    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b1a6fba-283e-4d58-8460-8ae01b6cb2a2-dshm" (OuterVolumeSpecName: "dshm") pod "9b1a6fba-283e-4d58-8460-8ae01b6cb2a2" (UID: "9b1a6fba-283e-4d58-8460-8ae01b6cb2a2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:55:24.528719 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:24.528542    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b1a6fba-283e-4d58-8460-8ae01b6cb2a2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9b1a6fba-283e-4d58-8460-8ae01b6cb2a2" (UID: "9b1a6fba-283e-4d58-8460-8ae01b6cb2a2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:55:24.627781 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:24.627738    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b1a6fba-283e-4d58-8460-8ae01b6cb2a2-tls-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 19:55:24.627781 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:24.627771    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b1a6fba-283e-4d58-8460-8ae01b6cb2a2-dshm\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 19:55:24.627781 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:24.627781    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ff78f\" (UniqueName: \"kubernetes.io/projected/9b1a6fba-283e-4d58-8460-8ae01b6cb2a2-kube-api-access-ff78f\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 19:55:25.025380 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:25.025335    2564 generic.go:356] "Generic (PLEG): container finished" podID="9b1a6fba-283e-4d58-8460-8ae01b6cb2a2" containerID="7710cb84a6c6cec5413a184cfeda416a77ac77b704674c3323ae11966437e71a" exitCode=0
Sep 18 19:55:25.025562 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:25.025405    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-sssbg"
Sep 18 19:55:25.025562 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:25.025425    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-sssbg" event={"ID":"9b1a6fba-283e-4d58-8460-8ae01b6cb2a2","Type":"ContainerDied","Data":"7710cb84a6c6cec5413a184cfeda416a77ac77b704674c3323ae11966437e71a"}
Sep 18 19:55:25.025562 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:25.025491    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-sssbg" event={"ID":"9b1a6fba-283e-4d58-8460-8ae01b6cb2a2","Type":"ContainerDied","Data":"cad248711f492eea175ce269cec6792e31a39efa990f735dbeb61cf0e9dd379d"}
Sep 18 19:55:25.025562 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:25.025521    2564 scope.go:117] "RemoveContainer" containerID="7710cb84a6c6cec5413a184cfeda416a77ac77b704674c3323ae11966437e71a"
Sep 18 19:55:25.036719 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:25.036700    2564 scope.go:117] "RemoveContainer" containerID="7710cb84a6c6cec5413a184cfeda416a77ac77b704674c3323ae11966437e71a"
Sep 18 19:55:25.037112 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:55:25.037089    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7710cb84a6c6cec5413a184cfeda416a77ac77b704674c3323ae11966437e71a\": container with ID starting with 7710cb84a6c6cec5413a184cfeda416a77ac77b704674c3323ae11966437e71a not found: ID does not exist" containerID="7710cb84a6c6cec5413a184cfeda416a77ac77b704674c3323ae11966437e71a"
Sep 18 19:55:25.037165 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:25.037122    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7710cb84a6c6cec5413a184cfeda416a77ac77b704674c3323ae11966437e71a"} err="failed to get container status \"7710cb84a6c6cec5413a184cfeda416a77ac77b704674c3323ae11966437e71a\": rpc error: code = NotFound desc = could not find container \"7710cb84a6c6cec5413a184cfeda416a77ac77b704674c3323ae11966437e71a\": container with ID starting with 7710cb84a6c6cec5413a184cfeda416a77ac77b704674c3323ae11966437e71a not found: ID does not exist"
Sep 18 19:55:25.048621 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:25.048587    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-sssbg"]
Sep 18 19:55:25.054920 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:25.054885    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-sssbg"]
Sep 18 19:55:25.999760 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:25.999723    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b1a6fba-283e-4d58-8460-8ae01b6cb2a2" path="/var/lib/kubelet/pods/9b1a6fba-283e-4d58-8460-8ae01b6cb2a2/volumes"
Sep 18 19:55:27.354576 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:27.354546    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dqmt8"
Sep 18 19:55:27.455880 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:27.455837    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dbc067a-fed0-4e78-85fe-cfb811d36662-istio-envoy\") pod \"5dbc067a-fed0-4e78-85fe-cfb811d36662\" (UID: \"5dbc067a-fed0-4e78-85fe-cfb811d36662\") "
Sep 18 19:55:27.456091 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:27.455895    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dbc067a-fed0-4e78-85fe-cfb811d36662-istio-token\") pod \"5dbc067a-fed0-4e78-85fe-cfb811d36662\" (UID: \"5dbc067a-fed0-4e78-85fe-cfb811d36662\") "
Sep 18 19:55:27.456091 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:27.455921    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dbc067a-fed0-4e78-85fe-cfb811d36662-credential-socket\") pod \"5dbc067a-fed0-4e78-85fe-cfb811d36662\" (UID: \"5dbc067a-fed0-4e78-85fe-cfb811d36662\") "
Sep 18 19:55:27.456091 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:27.455966    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-99ctq\" (UniqueName: \"kubernetes.io/projected/5dbc067a-fed0-4e78-85fe-cfb811d36662-kube-api-access-99ctq\") pod \"5dbc067a-fed0-4e78-85fe-cfb811d36662\" (UID: \"5dbc067a-fed0-4e78-85fe-cfb811d36662\") "
Sep 18 19:55:27.456091 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:27.456064    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dbc067a-fed0-4e78-85fe-cfb811d36662-workload-certs\") pod \"5dbc067a-fed0-4e78-85fe-cfb811d36662\" (UID: \"5dbc067a-fed0-4e78-85fe-cfb811d36662\") "
Sep 18 19:55:27.456306 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:27.456127    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dbc067a-fed0-4e78-85fe-cfb811d36662-istiod-ca-cert\") pod \"5dbc067a-fed0-4e78-85fe-cfb811d36662\" (UID: \"5dbc067a-fed0-4e78-85fe-cfb811d36662\") "
Sep 18 19:55:27.456306 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:27.456152    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dbc067a-fed0-4e78-85fe-cfb811d36662-workload-socket\") pod \"5dbc067a-fed0-4e78-85fe-cfb811d36662\" (UID: \"5dbc067a-fed0-4e78-85fe-cfb811d36662\") "
Sep 18 19:55:27.456306 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:27.456186    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dbc067a-fed0-4e78-85fe-cfb811d36662-istio-data\") pod \"5dbc067a-fed0-4e78-85fe-cfb811d36662\" (UID: \"5dbc067a-fed0-4e78-85fe-cfb811d36662\") "
Sep 18 19:55:27.456306 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:27.456204    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dbc067a-fed0-4e78-85fe-cfb811d36662-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5dbc067a-fed0-4e78-85fe-cfb811d36662" (UID: "5dbc067a-fed0-4e78-85fe-cfb811d36662"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:55:27.456306 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:27.456227    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dbc067a-fed0-4e78-85fe-cfb811d36662-istio-podinfo\") pod \"5dbc067a-fed0-4e78-85fe-cfb811d36662\" (UID: \"5dbc067a-fed0-4e78-85fe-cfb811d36662\") "
Sep 18 19:55:27.456553 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:27.456378    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dbc067a-fed0-4e78-85fe-cfb811d36662-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5dbc067a-fed0-4e78-85fe-cfb811d36662" (UID: "5dbc067a-fed0-4e78-85fe-cfb811d36662"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:55:27.456553 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:27.456492    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dbc067a-fed0-4e78-85fe-cfb811d36662-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5dbc067a-fed0-4e78-85fe-cfb811d36662" (UID: "5dbc067a-fed0-4e78-85fe-cfb811d36662"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:55:27.456678 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:27.456571    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dbc067a-fed0-4e78-85fe-cfb811d36662-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5dbc067a-fed0-4e78-85fe-cfb811d36662" (UID: "5dbc067a-fed0-4e78-85fe-cfb811d36662"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:55:27.456734 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:27.456710    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dbc067a-fed0-4e78-85fe-cfb811d36662-credential-socket\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 19:55:27.456789 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:27.456726    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dbc067a-fed0-4e78-85fe-cfb811d36662-istio-data" (OuterVolumeSpecName: "istio-data") pod "5dbc067a-fed0-4e78-85fe-cfb811d36662" (UID: "5dbc067a-fed0-4e78-85fe-cfb811d36662"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:55:27.456789 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:27.456731    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dbc067a-fed0-4e78-85fe-cfb811d36662-workload-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 19:55:27.456789 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:27.456770    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dbc067a-fed0-4e78-85fe-cfb811d36662-istiod-ca-cert\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 19:55:27.456789 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:27.456785    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dbc067a-fed0-4e78-85fe-cfb811d36662-workload-socket\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 19:55:27.458451 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:27.458418    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dbc067a-fed0-4e78-85fe-cfb811d36662-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5dbc067a-fed0-4e78-85fe-cfb811d36662" (UID: "5dbc067a-fed0-4e78-85fe-cfb811d36662"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:55:27.458451 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:27.458436    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dbc067a-fed0-4e78-85fe-cfb811d36662-kube-api-access-99ctq" (OuterVolumeSpecName: "kube-api-access-99ctq") pod "5dbc067a-fed0-4e78-85fe-cfb811d36662" (UID: "5dbc067a-fed0-4e78-85fe-cfb811d36662"). InnerVolumeSpecName "kube-api-access-99ctq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:55:27.458451 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:27.458442    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dbc067a-fed0-4e78-85fe-cfb811d36662-istio-token" (OuterVolumeSpecName: "istio-token") pod "5dbc067a-fed0-4e78-85fe-cfb811d36662" (UID: "5dbc067a-fed0-4e78-85fe-cfb811d36662"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:55:27.458641 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:27.458470    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5dbc067a-fed0-4e78-85fe-cfb811d36662-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5dbc067a-fed0-4e78-85fe-cfb811d36662" (UID: "5dbc067a-fed0-4e78-85fe-cfb811d36662"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 19:55:27.557672 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:27.557629    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dbc067a-fed0-4e78-85fe-cfb811d36662-istio-envoy\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 19:55:27.557672 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:27.557661    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dbc067a-fed0-4e78-85fe-cfb811d36662-istio-token\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 19:55:27.557672 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:27.557672    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99ctq\" (UniqueName: \"kubernetes.io/projected/5dbc067a-fed0-4e78-85fe-cfb811d36662-kube-api-access-99ctq\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 19:55:27.557672 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:27.557682    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dbc067a-fed0-4e78-85fe-cfb811d36662-istio-data\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 19:55:27.557954 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:27.557690    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dbc067a-fed0-4e78-85fe-cfb811d36662-istio-podinfo\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 19:55:28.045324 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:28.045280    2564 generic.go:356] "Generic (PLEG): container finished" podID="5dbc067a-fed0-4e78-85fe-cfb811d36662" containerID="25fa8c53daf79bb2b1c6c09325ffdff5b5aadb11aef6d94c140e3597c468ecf0" exitCode=0
Sep 18 19:55:28.045534 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:28.045358    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dqmt8"
Sep 18 19:55:28.045534 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:28.045382    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dqmt8" event={"ID":"5dbc067a-fed0-4e78-85fe-cfb811d36662","Type":"ContainerDied","Data":"25fa8c53daf79bb2b1c6c09325ffdff5b5aadb11aef6d94c140e3597c468ecf0"}
Sep 18 19:55:28.045534 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:28.045452    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dqmt8" event={"ID":"5dbc067a-fed0-4e78-85fe-cfb811d36662","Type":"ContainerDied","Data":"d4f39e03fb40011757c1e977a92c50973b2862f317b65aebc819d08628b72680"}
Sep 18 19:55:28.045534 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:28.045482    2564 scope.go:117] "RemoveContainer" containerID="25fa8c53daf79bb2b1c6c09325ffdff5b5aadb11aef6d94c140e3597c468ecf0"
Sep 18 19:55:28.054677 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:28.054654    2564 scope.go:117] "RemoveContainer" containerID="25fa8c53daf79bb2b1c6c09325ffdff5b5aadb11aef6d94c140e3597c468ecf0"
Sep 18 19:55:28.055073 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:55:28.055049    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25fa8c53daf79bb2b1c6c09325ffdff5b5aadb11aef6d94c140e3597c468ecf0\": container with ID starting with 25fa8c53daf79bb2b1c6c09325ffdff5b5aadb11aef6d94c140e3597c468ecf0 not found: ID does not exist" containerID="25fa8c53daf79bb2b1c6c09325ffdff5b5aadb11aef6d94c140e3597c468ecf0"
Sep 18 19:55:28.055185 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:28.055078    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25fa8c53daf79bb2b1c6c09325ffdff5b5aadb11aef6d94c140e3597c468ecf0"} err="failed to get container status \"25fa8c53daf79bb2b1c6c09325ffdff5b5aadb11aef6d94c140e3597c468ecf0\": rpc error: code = NotFound desc = could not find container \"25fa8c53daf79bb2b1c6c09325ffdff5b5aadb11aef6d94c140e3597c468ecf0\": container with ID starting with 25fa8c53daf79bb2b1c6c09325ffdff5b5aadb11aef6d94c140e3597c468ecf0 not found: ID does not exist"
Sep 18 19:55:28.062630 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:28.062601    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dqmt8"]
Sep 18 19:55:28.066472 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:28.066438    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dqmt8"]
Sep 18 19:55:29.875397 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:29.875361    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:55:30.001134 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:30.001092    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5dbc067a-fed0-4e78-85fe-cfb811d36662" path="/var/lib/kubelet/pods/5dbc067a-fed0-4e78-85fe-cfb811d36662/volumes"
Sep 18 19:55:30.143027 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:30.142965    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:55:34.203148 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:34.203111    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfznqrd"]
Sep 18 19:55:34.204567 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:34.204544    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b1a6fba-283e-4d58-8460-8ae01b6cb2a2" containerName="main"
Sep 18 19:55:34.204736 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:34.204725    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b1a6fba-283e-4d58-8460-8ae01b6cb2a2" containerName="main"
Sep 18 19:55:34.204824 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:34.204814    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dbc067a-fed0-4e78-85fe-cfb811d36662" containerName="istio-proxy"
Sep 18 19:55:34.204901 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:34.204892    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dbc067a-fed0-4e78-85fe-cfb811d36662" containerName="istio-proxy"
Sep 18 19:55:34.205092 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:34.205079    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b1a6fba-283e-4d58-8460-8ae01b6cb2a2" containerName="main"
Sep 18 19:55:34.205192 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:34.205183    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dbc067a-fed0-4e78-85fe-cfb811d36662" containerName="istio-proxy"
Sep 18 19:55:34.209168 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:34.209130    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfznqrd"
Sep 18 19:55:34.211479 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:34.211455    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Sep 18 19:55:34.211894 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:34.211875    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"default-dockercfg-mnsbk\""
Sep 18 19:55:34.212214 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:34.212198    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Sep 18 19:55:34.212871 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:34.212712    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs\""
Sep 18 19:55:34.216375 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:34.216347    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfznqrd"]
Sep 18 19:55:34.232233 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:34.232197    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:55:34.314257 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:34.314220    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c40ff6f-829c-4794-b761-f319751e8e3f-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfznqrd\" (UID: \"6c40ff6f-829c-4794-b761-f319751e8e3f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfznqrd"
Sep 18 19:55:34.314640 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:34.314277    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9mks\" (UniqueName: \"kubernetes.io/projected/6c40ff6f-829c-4794-b761-f319751e8e3f-kube-api-access-z9mks\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfznqrd\" (UID: \"6c40ff6f-829c-4794-b761-f319751e8e3f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfznqrd"
Sep 18 19:55:34.314640 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:34.314313    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c40ff6f-829c-4794-b761-f319751e8e3f-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfznqrd\" (UID: \"6c40ff6f-829c-4794-b761-f319751e8e3f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfznqrd"
Sep 18 19:55:34.314640 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:34.314384    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c40ff6f-829c-4794-b761-f319751e8e3f-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfznqrd\" (UID: \"6c40ff6f-829c-4794-b761-f319751e8e3f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfznqrd"
Sep 18 19:55:34.314640 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:34.314410    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c40ff6f-829c-4794-b761-f319751e8e3f-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfznqrd\" (UID: \"6c40ff6f-829c-4794-b761-f319751e8e3f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfznqrd"
Sep 18 19:55:34.314640 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:34.314428    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c40ff6f-829c-4794-b761-f319751e8e3f-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfznqrd\" (UID: \"6c40ff6f-829c-4794-b761-f319751e8e3f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfznqrd"
Sep 18 19:55:34.415633 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:34.415594    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c40ff6f-829c-4794-b761-f319751e8e3f-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfznqrd\" (UID: \"6c40ff6f-829c-4794-b761-f319751e8e3f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfznqrd"
Sep 18 19:55:34.415633 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:34.415649    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9mks\" (UniqueName: \"kubernetes.io/projected/6c40ff6f-829c-4794-b761-f319751e8e3f-kube-api-access-z9mks\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfznqrd\" (UID: \"6c40ff6f-829c-4794-b761-f319751e8e3f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfznqrd"
Sep 18 19:55:34.415904 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:34.415685    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c40ff6f-829c-4794-b761-f319751e8e3f-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfznqrd\" (UID: \"6c40ff6f-829c-4794-b761-f319751e8e3f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfznqrd"
Sep 18 19:55:34.415904 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:34.415735    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c40ff6f-829c-4794-b761-f319751e8e3f-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfznqrd\" (UID: \"6c40ff6f-829c-4794-b761-f319751e8e3f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfznqrd"
Sep 18 19:55:34.415904 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:34.415771    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c40ff6f-829c-4794-b761-f319751e8e3f-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfznqrd\" (UID: \"6c40ff6f-829c-4794-b761-f319751e8e3f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfznqrd"
Sep 18 19:55:34.416119 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:34.415899    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c40ff6f-829c-4794-b761-f319751e8e3f-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfznqrd\" (UID: \"6c40ff6f-829c-4794-b761-f319751e8e3f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfznqrd"
Sep 18 19:55:34.416178 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:34.416118    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c40ff6f-829c-4794-b761-f319751e8e3f-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfznqrd\" (UID: \"6c40ff6f-829c-4794-b761-f319751e8e3f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfznqrd"
Sep 18 19:55:34.416234 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:34.416195    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c40ff6f-829c-4794-b761-f319751e8e3f-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfznqrd\" (UID: \"6c40ff6f-829c-4794-b761-f319751e8e3f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfznqrd"
Sep 18 19:55:34.416234 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:34.416205    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c40ff6f-829c-4794-b761-f319751e8e3f-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfznqrd\" (UID: \"6c40ff6f-829c-4794-b761-f319751e8e3f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfznqrd"
Sep 18 19:55:34.418220 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:34.418197    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c40ff6f-829c-4794-b761-f319751e8e3f-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfznqrd\" (UID: \"6c40ff6f-829c-4794-b761-f319751e8e3f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfznqrd"
Sep 18 19:55:34.418428 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:34.418411    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c40ff6f-829c-4794-b761-f319751e8e3f-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfznqrd\" (UID: \"6c40ff6f-829c-4794-b761-f319751e8e3f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfznqrd"
Sep 18 19:55:34.422665 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:34.422640    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9mks\" (UniqueName: \"kubernetes.io/projected/6c40ff6f-829c-4794-b761-f319751e8e3f-kube-api-access-z9mks\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfznqrd\" (UID: \"6c40ff6f-829c-4794-b761-f319751e8e3f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfznqrd"
Sep 18 19:55:34.526650 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:34.526560    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfznqrd"
Sep 18 19:55:34.808478 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:34.808451    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfznqrd"]
Sep 18 19:55:34.810712 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:55:34.810678    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c40ff6f_829c_4794_b761_f319751e8e3f.slice/crio-07f2f3f537222fd94eb8d3157a321c17e876c6f968ac6645c28b1fe48f7c481e WatchSource:0}: Error finding container 07f2f3f537222fd94eb8d3157a321c17e876c6f968ac6645c28b1fe48f7c481e: Status 404 returned error can't find the container with id 07f2f3f537222fd94eb8d3157a321c17e876c6f968ac6645c28b1fe48f7c481e
Sep 18 19:55:35.082886 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:35.082801    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfznqrd" event={"ID":"6c40ff6f-829c-4794-b761-f319751e8e3f","Type":"ContainerStarted","Data":"51b63958a5b017933e41dc60f3baed6afc5b94111d9847ad30863f778708a9e9"}
Sep 18 19:55:35.082886 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:35.082852    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfznqrd" event={"ID":"6c40ff6f-829c-4794-b761-f319751e8e3f","Type":"ContainerStarted","Data":"07f2f3f537222fd94eb8d3157a321c17e876c6f968ac6645c28b1fe48f7c481e"}
Sep 18 19:55:35.099081 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:35.099031    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfznqrd" podStartSLOduration=1.099012892 podStartE2EDuration="1.099012892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:55:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:55:35.097248769 +0000 UTC m=+1051.607604888" watchObservedRunningTime="2026-09-18 19:55:35.099012892 +0000 UTC m=+1051.609368974"
Sep 18 19:55:35.527439 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:35.527405    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfznqrd"
Sep 18 19:55:36.537605 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:36.537570    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:55:36.635416 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:36.635383    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:55:37.887824 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:37.887790    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfznqrd"]
Sep 18 19:55:37.897507 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:37.897475    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:55:38.108133 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:55:38.108099    2564 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 19:55:38.108133 ip-10-0-128-161 kubenswrapper[2564]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:55:38.108133 ip-10-0-128-161 kubenswrapper[2564]:         , stdout: , stderr: , exit code -1
Sep 18 19:55:38.108133 ip-10-0-128-161 kubenswrapper[2564]:  > containerID="51b63958a5b017933e41dc60f3baed6afc5b94111d9847ad30863f778708a9e9" cmd=["/bin/sleep","15"]
Sep 18 19:55:38.108375 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:38.108148    2564 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfznqrd" message="" err=<
Sep 18 19:55:38.108375 ip-10-0-128-161 kubenswrapper[2564]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:55:38.108375 ip-10-0-128-161 kubenswrapper[2564]:         , stdout: , stderr: , exit code -1
Sep 18 19:55:38.108375 ip-10-0-128-161 kubenswrapper[2564]:  >
Sep 18 19:55:38.108375 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:55:38.108161    2564 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 19:55:38.108375 ip-10-0-128-161 kubenswrapper[2564]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:55:38.108375 ip-10-0-128-161 kubenswrapper[2564]:         , stdout: , stderr: , exit code -1
Sep 18 19:55:38.108375 ip-10-0-128-161 kubenswrapper[2564]:  > pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfznqrd" podUID="6c40ff6f-829c-4794-b761-f319751e8e3f" containerName="main" containerID="cri-o://51b63958a5b017933e41dc60f3baed6afc5b94111d9847ad30863f778708a9e9"
Sep 18 19:55:38.108375 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:38.108212    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfznqrd" podUID="6c40ff6f-829c-4794-b761-f319751e8e3f" containerName="main" containerID="cri-o://51b63958a5b017933e41dc60f3baed6afc5b94111d9847ad30863f778708a9e9" gracePeriod=60
Sep 18 19:55:38.366665 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:38.366631    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfznqrd"
Sep 18 19:55:38.453081 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:38.453044    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c40ff6f-829c-4794-b761-f319751e8e3f-tls-certs\") pod \"6c40ff6f-829c-4794-b761-f319751e8e3f\" (UID: \"6c40ff6f-829c-4794-b761-f319751e8e3f\") "
Sep 18 19:55:38.453299 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:38.453110    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z9mks\" (UniqueName: \"kubernetes.io/projected/6c40ff6f-829c-4794-b761-f319751e8e3f-kube-api-access-z9mks\") pod \"6c40ff6f-829c-4794-b761-f319751e8e3f\" (UID: \"6c40ff6f-829c-4794-b761-f319751e8e3f\") "
Sep 18 19:55:38.453299 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:38.453158    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c40ff6f-829c-4794-b761-f319751e8e3f-home\") pod \"6c40ff6f-829c-4794-b761-f319751e8e3f\" (UID: \"6c40ff6f-829c-4794-b761-f319751e8e3f\") "
Sep 18 19:55:38.453299 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:38.453263    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c40ff6f-829c-4794-b761-f319751e8e3f-tmp-dir\") pod \"6c40ff6f-829c-4794-b761-f319751e8e3f\" (UID: \"6c40ff6f-829c-4794-b761-f319751e8e3f\") "
Sep 18 19:55:38.453483 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:38.453312    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c40ff6f-829c-4794-b761-f319751e8e3f-dshm\") pod \"6c40ff6f-829c-4794-b761-f319751e8e3f\" (UID: \"6c40ff6f-829c-4794-b761-f319751e8e3f\") "
Sep 18 19:55:38.453483 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:38.453379    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c40ff6f-829c-4794-b761-f319751e8e3f-model-cache\") pod \"6c40ff6f-829c-4794-b761-f319751e8e3f\" (UID: \"6c40ff6f-829c-4794-b761-f319751e8e3f\") "
Sep 18 19:55:38.453483 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:38.453445    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c40ff6f-829c-4794-b761-f319751e8e3f-home" (OuterVolumeSpecName: "home") pod "6c40ff6f-829c-4794-b761-f319751e8e3f" (UID: "6c40ff6f-829c-4794-b761-f319751e8e3f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:55:38.453622 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:38.453512    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c40ff6f-829c-4794-b761-f319751e8e3f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6c40ff6f-829c-4794-b761-f319751e8e3f" (UID: "6c40ff6f-829c-4794-b761-f319751e8e3f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:55:38.453662 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:38.453645    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c40ff6f-829c-4794-b761-f319751e8e3f-model-cache" (OuterVolumeSpecName: "model-cache") pod "6c40ff6f-829c-4794-b761-f319751e8e3f" (UID: "6c40ff6f-829c-4794-b761-f319751e8e3f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:55:38.453783 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:38.453764    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c40ff6f-829c-4794-b761-f319751e8e3f-model-cache\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 19:55:38.453822 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:38.453788    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c40ff6f-829c-4794-b761-f319751e8e3f-home\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 19:55:38.453822 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:38.453801    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c40ff6f-829c-4794-b761-f319751e8e3f-tmp-dir\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 19:55:38.455550 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:38.455522    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c40ff6f-829c-4794-b761-f319751e8e3f-dshm" (OuterVolumeSpecName: "dshm") pod "6c40ff6f-829c-4794-b761-f319751e8e3f" (UID: "6c40ff6f-829c-4794-b761-f319751e8e3f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:55:38.455550 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:38.455539    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c40ff6f-829c-4794-b761-f319751e8e3f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6c40ff6f-829c-4794-b761-f319751e8e3f" (UID: "6c40ff6f-829c-4794-b761-f319751e8e3f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:55:38.455718 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:38.455556    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c40ff6f-829c-4794-b761-f319751e8e3f-kube-api-access-z9mks" (OuterVolumeSpecName: "kube-api-access-z9mks") pod "6c40ff6f-829c-4794-b761-f319751e8e3f" (UID: "6c40ff6f-829c-4794-b761-f319751e8e3f"). InnerVolumeSpecName "kube-api-access-z9mks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:55:38.554555 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:38.554474    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c40ff6f-829c-4794-b761-f319751e8e3f-dshm\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 19:55:38.554555 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:38.554505    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c40ff6f-829c-4794-b761-f319751e8e3f-tls-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 19:55:38.554555 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:38.554517    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9mks\" (UniqueName: \"kubernetes.io/projected/6c40ff6f-829c-4794-b761-f319751e8e3f-kube-api-access-z9mks\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 19:55:39.105284 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:39.105247    2564 generic.go:356] "Generic (PLEG): container finished" podID="6c40ff6f-829c-4794-b761-f319751e8e3f" containerID="51b63958a5b017933e41dc60f3baed6afc5b94111d9847ad30863f778708a9e9" exitCode=0
Sep 18 19:55:39.105750 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:39.105294    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfznqrd" event={"ID":"6c40ff6f-829c-4794-b761-f319751e8e3f","Type":"ContainerDied","Data":"51b63958a5b017933e41dc60f3baed6afc5b94111d9847ad30863f778708a9e9"}
Sep 18 19:55:39.105750 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:39.105301    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfznqrd"
Sep 18 19:55:39.105750 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:39.105355    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfznqrd" event={"ID":"6c40ff6f-829c-4794-b761-f319751e8e3f","Type":"ContainerDied","Data":"07f2f3f537222fd94eb8d3157a321c17e876c6f968ac6645c28b1fe48f7c481e"}
Sep 18 19:55:39.105750 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:39.105388    2564 scope.go:117] "RemoveContainer" containerID="51b63958a5b017933e41dc60f3baed6afc5b94111d9847ad30863f778708a9e9"
Sep 18 19:55:39.115456 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:39.115319    2564 scope.go:117] "RemoveContainer" containerID="51b63958a5b017933e41dc60f3baed6afc5b94111d9847ad30863f778708a9e9"
Sep 18 19:55:39.115695 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:55:39.115674    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51b63958a5b017933e41dc60f3baed6afc5b94111d9847ad30863f778708a9e9\": container with ID starting with 51b63958a5b017933e41dc60f3baed6afc5b94111d9847ad30863f778708a9e9 not found: ID does not exist" containerID="51b63958a5b017933e41dc60f3baed6afc5b94111d9847ad30863f778708a9e9"
Sep 18 19:55:39.115753 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:39.115706    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51b63958a5b017933e41dc60f3baed6afc5b94111d9847ad30863f778708a9e9"} err="failed to get container status \"51b63958a5b017933e41dc60f3baed6afc5b94111d9847ad30863f778708a9e9\": rpc error: code = NotFound desc = could not find container \"51b63958a5b017933e41dc60f3baed6afc5b94111d9847ad30863f778708a9e9\": container with ID starting with 51b63958a5b017933e41dc60f3baed6afc5b94111d9847ad30863f778708a9e9 not found: ID does not exist"
Sep 18 19:55:39.126524 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:39.126495    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfznqrd"]
Sep 18 19:55:39.130364 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:39.130335    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfznqrd"]
Sep 18 19:55:40.001024 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:40.000970    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c40ff6f-829c-4794-b761-f319751e8e3f" path="/var/lib/kubelet/pods/6c40ff6f-829c-4794-b761-f319751e8e3f/volumes"
Sep 18 19:55:41.188331 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:41.188276    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:55:41.563694 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:41.563603    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:55:44.748253 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:44.748162    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:55:44.845668 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:44.845631    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:55:46.883496 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:46.883463    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:55:47.517044 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:47.517011    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:55:48.251693 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:48.251657    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:55:48.431628 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:48.431587    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-wwvlm"]
Sep 18 19:55:48.432237 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:48.432215    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c40ff6f-829c-4794-b761-f319751e8e3f" containerName="main"
Sep 18 19:55:48.432237 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:48.432239    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c40ff6f-829c-4794-b761-f319751e8e3f" containerName="main"
Sep 18 19:55:48.432405 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:48.432343    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c40ff6f-829c-4794-b761-f319751e8e3f" containerName="main"
Sep 18 19:55:48.436921 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:48.436893    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-wwvlm"
Sep 18 19:55:48.439916 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:48.439890    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Sep 18 19:55:48.440129 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:48.440027    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Sep 18 19:55:48.440129 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:48.440043    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Sep 18 19:55:48.440260 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:48.440151    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-wgnmt\""
Sep 18 19:55:48.440315 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:48.440267    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-epp-sa-dockercfg-gjlv5\""
Sep 18 19:55:48.443778 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:48.443751    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-wwvlm"]
Sep 18 19:55:48.463860 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:48.463833    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:55:48.541619 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:48.541519    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv2n2\" (UniqueName: \"kubernetes.io/projected/5fb2a776-c901-46d5-a31c-cbbc1ef1f9b0-kube-api-access-zv2n2\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-wwvlm\" (UID: \"5fb2a776-c901-46d5-a31c-cbbc1ef1f9b0\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-wwvlm"
Sep 18 19:55:48.541928 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:48.541895    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fb2a776-c901-46d5-a31c-cbbc1ef1f9b0-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-wwvlm\" (UID: \"5fb2a776-c901-46d5-a31c-cbbc1ef1f9b0\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-wwvlm"
Sep 18 19:55:48.643513 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:48.643462    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv2n2\" (UniqueName: \"kubernetes.io/projected/5fb2a776-c901-46d5-a31c-cbbc1ef1f9b0-kube-api-access-zv2n2\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-wwvlm\" (UID: \"5fb2a776-c901-46d5-a31c-cbbc1ef1f9b0\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-wwvlm"
Sep 18 19:55:48.643733 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:48.643558    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fb2a776-c901-46d5-a31c-cbbc1ef1f9b0-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-wwvlm\" (UID: \"5fb2a776-c901-46d5-a31c-cbbc1ef1f9b0\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-wwvlm"
Sep 18 19:55:48.646165 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:48.646131    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fb2a776-c901-46d5-a31c-cbbc1ef1f9b0-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-wwvlm\" (UID: \"5fb2a776-c901-46d5-a31c-cbbc1ef1f9b0\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-wwvlm"
Sep 18 19:55:48.651645 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:48.651613    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv2n2\" (UniqueName: \"kubernetes.io/projected/5fb2a776-c901-46d5-a31c-cbbc1ef1f9b0-kube-api-access-zv2n2\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-wwvlm\" (UID: \"5fb2a776-c901-46d5-a31c-cbbc1ef1f9b0\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-wwvlm"
Sep 18 19:55:48.750388 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:48.750346    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-wwvlm"
Sep 18 19:55:49.024048 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:49.024019    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-wwvlm"]
Sep 18 19:55:49.026714 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:55:49.026685    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fb2a776_c901_46d5_a31c_cbbc1ef1f9b0.slice/crio-44e8bfc789b5c0bcfa140f0e57c8a168c0fdf26e014324480d390e31bbb9a24f WatchSource:0}: Error finding container 44e8bfc789b5c0bcfa140f0e57c8a168c0fdf26e014324480d390e31bbb9a24f: Status 404 returned error can't find the container with id 44e8bfc789b5c0bcfa140f0e57c8a168c0fdf26e014324480d390e31bbb9a24f
Sep 18 19:55:49.158542 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:49.158499    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-wwvlm" event={"ID":"5fb2a776-c901-46d5-a31c-cbbc1ef1f9b0","Type":"ContainerStarted","Data":"3d266c91651dca660b0aa9987c1e3c84794048fdbb3dd77b80397d6b93cfe208"}
Sep 18 19:55:49.158726 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:49.158572    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-wwvlm" event={"ID":"5fb2a776-c901-46d5-a31c-cbbc1ef1f9b0","Type":"ContainerStarted","Data":"44e8bfc789b5c0bcfa140f0e57c8a168c0fdf26e014324480d390e31bbb9a24f"}
Sep 18 19:55:49.158726 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:49.158608    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-wwvlm"
Sep 18 19:55:49.177377 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:49.177296    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-wwvlm" podStartSLOduration=1.177276441 podStartE2EDuration="1.177276441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:55:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:55:49.175600119 +0000 UTC m=+1065.685956236" watchObservedRunningTime="2026-09-18 19:55:49.177276441 +0000 UTC m=+1065.687632546"
Sep 18 19:55:53.308163 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:55:53.308126    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:56:01.540403 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:56:01.540356    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:56:02.075941 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:56:02.075859    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:56:02.177509 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:56:02.177469    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:56:03.213552 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:56:03.213505    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:56:03.222633 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:56:03.222596    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche74hft"]
Sep 18 19:56:03.310393 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:56:03.310357    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:56:11.661278 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:56:11.661233    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche74hft" podUID="cd99db9e-1358-481a-812e-dcf9aed08100" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:56:18.243999 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:56:18.243915    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche74hft" podUID="cd99db9e-1358-481a-812e-dcf9aed08100" containerName="main" containerID="cri-o://8b19856456e0e9ad9516dabb262fba8bd02191aa7b0e8d02bd7c0219109bc650" gracePeriod=45
Sep 18 19:56:20.165904 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:56:20.165874    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-wwvlm"
Sep 18 19:56:21.661133 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:56:21.661097    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche74hft" podUID="cd99db9e-1358-481a-812e-dcf9aed08100" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:56:31.661410 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:56:31.661364    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche74hft" podUID="cd99db9e-1358-481a-812e-dcf9aed08100" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:56:31.661876 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:56:31.661471    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche74hft"
Sep 18 19:56:41.660911 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:56:41.660861    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche74hft" podUID="cd99db9e-1358-481a-812e-dcf9aed08100" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:56:48.459266 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:56:48.459233    2564 generic.go:356] "Generic (PLEG): container finished" podID="cd99db9e-1358-481a-812e-dcf9aed08100" containerID="8b19856456e0e9ad9516dabb262fba8bd02191aa7b0e8d02bd7c0219109bc650" exitCode=0
Sep 18 19:56:48.459641 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:56:48.459295    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche74hft" event={"ID":"cd99db9e-1358-481a-812e-dcf9aed08100","Type":"ContainerDied","Data":"8b19856456e0e9ad9516dabb262fba8bd02191aa7b0e8d02bd7c0219109bc650"}
Sep 18 19:56:48.502670 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:56:48.502599    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche74hft"
Sep 18 19:56:48.686457 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:56:48.686422    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lp8n5\" (UniqueName: \"kubernetes.io/projected/cd99db9e-1358-481a-812e-dcf9aed08100-kube-api-access-lp8n5\") pod \"cd99db9e-1358-481a-812e-dcf9aed08100\" (UID: \"cd99db9e-1358-481a-812e-dcf9aed08100\") "
Sep 18 19:56:48.686633 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:56:48.686510    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd99db9e-1358-481a-812e-dcf9aed08100-tls-certs\") pod \"cd99db9e-1358-481a-812e-dcf9aed08100\" (UID: \"cd99db9e-1358-481a-812e-dcf9aed08100\") "
Sep 18 19:56:48.688664 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:56:48.688626    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd99db9e-1358-481a-812e-dcf9aed08100-kube-api-access-lp8n5" (OuterVolumeSpecName: "kube-api-access-lp8n5") pod "cd99db9e-1358-481a-812e-dcf9aed08100" (UID: "cd99db9e-1358-481a-812e-dcf9aed08100"). InnerVolumeSpecName "kube-api-access-lp8n5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:56:48.688787 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:56:48.688696    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd99db9e-1358-481a-812e-dcf9aed08100-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cd99db9e-1358-481a-812e-dcf9aed08100" (UID: "cd99db9e-1358-481a-812e-dcf9aed08100"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:56:48.787518 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:56:48.787429    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd99db9e-1358-481a-812e-dcf9aed08100-tls-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 19:56:48.787518 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:56:48.787464    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp8n5\" (UniqueName: \"kubernetes.io/projected/cd99db9e-1358-481a-812e-dcf9aed08100-kube-api-access-lp8n5\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 19:56:49.467765 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:56:49.467731    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche74hft"
Sep 18 19:56:49.468434 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:56:49.467756    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche74hft" event={"ID":"cd99db9e-1358-481a-812e-dcf9aed08100","Type":"ContainerDied","Data":"68778343d6e1c5ee53b1a4e4ad3d88cb23c01d9933129d4f08fbac68bf1a1108"}
Sep 18 19:56:49.468434 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:56:49.467841    2564 scope.go:117] "RemoveContainer" containerID="8b19856456e0e9ad9516dabb262fba8bd02191aa7b0e8d02bd7c0219109bc650"
Sep 18 19:56:49.488548 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:56:49.488512    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche74hft"]
Sep 18 19:56:49.492211 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:56:49.492180    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche74hft"]
Sep 18 19:56:50.000644 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:56:50.000610    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd99db9e-1358-481a-812e-dcf9aed08100" path="/var/lib/kubelet/pods/cd99db9e-1358-481a-812e-dcf9aed08100/volumes"
Sep 18 19:56:51.380153 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:56:51.380111    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:56:51.477763 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:56:51.477727    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:56:55.449716 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:56:55.449674    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:56:57.477497 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:56:57.477461    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:57:00.635635 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:57:00.635589    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:57:00.851270 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:57:00.851234    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbkdkzc"]
Sep 18 19:57:00.851618 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:57:00.851606    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd99db9e-1358-481a-812e-dcf9aed08100" containerName="main"
Sep 18 19:57:00.851669 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:57:00.851620    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd99db9e-1358-481a-812e-dcf9aed08100" containerName="main"
Sep 18 19:57:00.851704 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:57:00.851690    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd99db9e-1358-481a-812e-dcf9aed08100" containerName="main"
Sep 18 19:57:00.856482 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:57:00.856455    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbkdkzc"
Sep 18 19:57:00.859202 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:57:00.859173    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Sep 18 19:57:00.859356 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:57:00.859340    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-epp-sa-dockercfg-6fcjm\""
Sep 18 19:57:00.859449 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:57:00.859430    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Sep 18 19:57:00.859504 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:57:00.859436    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Sep 18 19:57:00.859565 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:57:00.859440    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-kd85p\""
Sep 18 19:57:00.871001 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:57:00.869413    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbkdkzc"]
Sep 18 19:57:00.895891 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:57:00.895788    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfl26\" (UniqueName: \"kubernetes.io/projected/e97a389b-c8ec-4a4c-b508-443816da8bc1-kube-api-access-pfl26\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbkdkzc\" (UID: \"e97a389b-c8ec-4a4c-b508-443816da8bc1\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbkdkzc"
Sep 18 19:57:00.895891 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:57:00.895852    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e97a389b-c8ec-4a4c-b508-443816da8bc1-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbkdkzc\" (UID: \"e97a389b-c8ec-4a4c-b508-443816da8bc1\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbkdkzc"
Sep 18 19:57:00.913175 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:57:00.913143    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:57:00.997482 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:57:00.997442    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfl26\" (UniqueName: \"kubernetes.io/projected/e97a389b-c8ec-4a4c-b508-443816da8bc1-kube-api-access-pfl26\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbkdkzc\" (UID: \"e97a389b-c8ec-4a4c-b508-443816da8bc1\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbkdkzc"
Sep 18 19:57:00.997674 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:57:00.997500    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e97a389b-c8ec-4a4c-b508-443816da8bc1-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbkdkzc\" (UID: \"e97a389b-c8ec-4a4c-b508-443816da8bc1\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbkdkzc"
Sep 18 19:57:01.000102 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:57:01.000070    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e97a389b-c8ec-4a4c-b508-443816da8bc1-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbkdkzc\" (UID: \"e97a389b-c8ec-4a4c-b508-443816da8bc1\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbkdkzc"
Sep 18 19:57:01.004953 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:57:01.004928    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfl26\" (UniqueName: \"kubernetes.io/projected/e97a389b-c8ec-4a4c-b508-443816da8bc1-kube-api-access-pfl26\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbkdkzc\" (UID: \"e97a389b-c8ec-4a4c-b508-443816da8bc1\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbkdkzc"
Sep 18 19:57:01.170141 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:57:01.170042    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbkdkzc"
Sep 18 19:57:01.267785 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:57:01.267735    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:57:01.444874 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:57:01.444784    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbkdkzc"]
Sep 18 19:57:01.447234 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:57:01.447202    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode97a389b_c8ec_4a4c_b508_443816da8bc1.slice/crio-3b56aa6cb2322339903a8a0b5d7bbf52c22ff0ca53d86321bcbd9c87359299c0 WatchSource:0}: Error finding container 3b56aa6cb2322339903a8a0b5d7bbf52c22ff0ca53d86321bcbd9c87359299c0: Status 404 returned error can't find the container with id 3b56aa6cb2322339903a8a0b5d7bbf52c22ff0ca53d86321bcbd9c87359299c0
Sep 18 19:57:01.527340 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:57:01.527293    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbkdkzc" event={"ID":"e97a389b-c8ec-4a4c-b508-443816da8bc1","Type":"ContainerStarted","Data":"bbd444132eacbd69a2629a60f1983d94691bc1c10eac13fd66d8f6fe0f283eb4"}
Sep 18 19:57:01.527553 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:57:01.527367    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbkdkzc" event={"ID":"e97a389b-c8ec-4a4c-b508-443816da8bc1","Type":"ContainerStarted","Data":"3b56aa6cb2322339903a8a0b5d7bbf52c22ff0ca53d86321bcbd9c87359299c0"}
Sep 18 19:57:01.527553 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:57:01.527398    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbkdkzc"
Sep 18 19:57:01.540782 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:57:01.540717    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbkdkzc" podStartSLOduration=1.540670322 podStartE2EDuration="1.540670322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:57:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:57:01.54049014 +0000 UTC m=+1138.050846271" watchObservedRunningTime="2026-09-18 19:57:01.540670322 +0000 UTC m=+1138.051026415"
Sep 18 19:57:03.158654 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:57:03.158613    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:57:03.259058 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:57:03.259025    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:57:22.491543 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:57:22.491506    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:57:30.701554 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:57:30.701470    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:57:32.534504 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:57:32.534472    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbkdkzc"
Sep 18 19:57:33.935116 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:57:33.935078    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbkdkzc"]
Sep 18 19:57:33.953950 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:57:33.953893    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:57:34.052948 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:57:34.052913    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:57:42.534841 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:57:42.534803    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbkdkzc" podUID="e97a389b-c8ec-4a4c-b508-443816da8bc1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:57:48.964551 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:57:48.964489    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbkdkzc" podUID="e97a389b-c8ec-4a4c-b508-443816da8bc1" containerName="main" containerID="cri-o://bbd444132eacbd69a2629a60f1983d94691bc1c10eac13fd66d8f6fe0f283eb4" gracePeriod=45
Sep 18 19:57:52.534293 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:57:52.534256    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbkdkzc" podUID="e97a389b-c8ec-4a4c-b508-443816da8bc1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:57:53.832399 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:57:53.832356    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-wwvlm"]
Sep 18 19:57:53.838742 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:57:53.838689    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:57:53.938145 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:57:53.938106    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:00.165906 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:00.165863    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-wwvlm" podUID="5fb2a776-c901-46d5-a31c-cbbc1ef1f9b0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:58:02.534836 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:02.534791    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbkdkzc" podUID="e97a389b-c8ec-4a4c-b508-443816da8bc1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:58:02.535264 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:02.534896    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbkdkzc"
Sep 18 19:58:03.973854 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:03.973824    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-58vl4_945a444e-9381-4e2f-b4c1-64be0ec88e6c/console-operator/2.log"
Sep 18 19:58:03.975865 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:03.975845    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-58vl4_945a444e-9381-4e2f-b4c1-64be0ec88e6c/console-operator/2.log"
Sep 18 19:58:08.857369 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:08.857316    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-wwvlm" podUID="5fb2a776-c901-46d5-a31c-cbbc1ef1f9b0" containerName="main" containerID="cri-o://3d266c91651dca660b0aa9987c1e3c84794048fdbb3dd77b80397d6b93cfe208" gracePeriod=45
Sep 18 19:58:10.165377 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:10.165330    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-wwvlm" podUID="5fb2a776-c901-46d5-a31c-cbbc1ef1f9b0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:58:12.534478 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:12.534435    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbkdkzc" podUID="e97a389b-c8ec-4a4c-b508-443816da8bc1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:58:19.221641 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:19.221594    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbkdkzc"
Sep 18 19:58:19.298667 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:19.298585    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e97a389b-c8ec-4a4c-b508-443816da8bc1-tls-certs\") pod \"e97a389b-c8ec-4a4c-b508-443816da8bc1\" (UID: \"e97a389b-c8ec-4a4c-b508-443816da8bc1\") "
Sep 18 19:58:19.298667 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:19.298632    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pfl26\" (UniqueName: \"kubernetes.io/projected/e97a389b-c8ec-4a4c-b508-443816da8bc1-kube-api-access-pfl26\") pod \"e97a389b-c8ec-4a4c-b508-443816da8bc1\" (UID: \"e97a389b-c8ec-4a4c-b508-443816da8bc1\") "
Sep 18 19:58:19.300869 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:19.300837    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e97a389b-c8ec-4a4c-b508-443816da8bc1-kube-api-access-pfl26" (OuterVolumeSpecName: "kube-api-access-pfl26") pod "e97a389b-c8ec-4a4c-b508-443816da8bc1" (UID: "e97a389b-c8ec-4a4c-b508-443816da8bc1"). InnerVolumeSpecName "kube-api-access-pfl26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:58:19.301026 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:19.300921    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e97a389b-c8ec-4a4c-b508-443816da8bc1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e97a389b-c8ec-4a4c-b508-443816da8bc1" (UID: "e97a389b-c8ec-4a4c-b508-443816da8bc1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:58:19.399603 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:19.399561    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e97a389b-c8ec-4a4c-b508-443816da8bc1-tls-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 19:58:19.399603 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:19.399596    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfl26\" (UniqueName: \"kubernetes.io/projected/e97a389b-c8ec-4a4c-b508-443816da8bc1-kube-api-access-pfl26\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 19:58:19.921959 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:19.921924    2564 generic.go:356] "Generic (PLEG): container finished" podID="e97a389b-c8ec-4a4c-b508-443816da8bc1" containerID="bbd444132eacbd69a2629a60f1983d94691bc1c10eac13fd66d8f6fe0f283eb4" exitCode=0
Sep 18 19:58:19.922249 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:19.921976    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbkdkzc"
Sep 18 19:58:19.922249 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:19.922009    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbkdkzc" event={"ID":"e97a389b-c8ec-4a4c-b508-443816da8bc1","Type":"ContainerDied","Data":"bbd444132eacbd69a2629a60f1983d94691bc1c10eac13fd66d8f6fe0f283eb4"}
Sep 18 19:58:19.922249 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:19.922066    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbkdkzc" event={"ID":"e97a389b-c8ec-4a4c-b508-443816da8bc1","Type":"ContainerDied","Data":"3b56aa6cb2322339903a8a0b5d7bbf52c22ff0ca53d86321bcbd9c87359299c0"}
Sep 18 19:58:19.922249 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:19.922091    2564 scope.go:117] "RemoveContainer" containerID="bbd444132eacbd69a2629a60f1983d94691bc1c10eac13fd66d8f6fe0f283eb4"
Sep 18 19:58:19.937676 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:19.937651    2564 scope.go:117] "RemoveContainer" containerID="bbd444132eacbd69a2629a60f1983d94691bc1c10eac13fd66d8f6fe0f283eb4"
Sep 18 19:58:19.937975 ip-10-0-128-161 kubenswrapper[2564]: E0918 19:58:19.937954    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbd444132eacbd69a2629a60f1983d94691bc1c10eac13fd66d8f6fe0f283eb4\": container with ID starting with bbd444132eacbd69a2629a60f1983d94691bc1c10eac13fd66d8f6fe0f283eb4 not found: ID does not exist" containerID="bbd444132eacbd69a2629a60f1983d94691bc1c10eac13fd66d8f6fe0f283eb4"
Sep 18 19:58:19.938044 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:19.938000    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbd444132eacbd69a2629a60f1983d94691bc1c10eac13fd66d8f6fe0f283eb4"} err="failed to get container status \"bbd444132eacbd69a2629a60f1983d94691bc1c10eac13fd66d8f6fe0f283eb4\": rpc error: code = NotFound desc = could not find container \"bbd444132eacbd69a2629a60f1983d94691bc1c10eac13fd66d8f6fe0f283eb4\": container with ID starting with bbd444132eacbd69a2629a60f1983d94691bc1c10eac13fd66d8f6fe0f283eb4 not found: ID does not exist"
Sep 18 19:58:19.949308 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:19.949275    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbkdkzc"]
Sep 18 19:58:19.954777 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:19.954750    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbkdkzc"]
Sep 18 19:58:20.001004 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:20.000948    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e97a389b-c8ec-4a4c-b508-443816da8bc1" path="/var/lib/kubelet/pods/e97a389b-c8ec-4a4c-b508-443816da8bc1/volumes"
Sep 18 19:58:20.165942 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:20.165899    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-wwvlm" podUID="5fb2a776-c901-46d5-a31c-cbbc1ef1f9b0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:58:20.166134 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:20.166041    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-wwvlm"
Sep 18 19:58:23.593205 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:23.593169    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:23.692180 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:23.692145    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:29.644009 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:29.643941    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:30.165939 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:30.165901    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-wwvlm" podUID="5fb2a776-c901-46d5-a31c-cbbc1ef1f9b0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:58:35.088007 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:35.087942    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:35.189013 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:35.188943    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:35.533266 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:35.533229    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:35.757330 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:35.757295    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche2nbmz"]
Sep 18 19:58:35.757701 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:35.757683    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e97a389b-c8ec-4a4c-b508-443816da8bc1" containerName="main"
Sep 18 19:58:35.757752 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:35.757705    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e97a389b-c8ec-4a4c-b508-443816da8bc1" containerName="main"
Sep 18 19:58:35.757847 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:35.757798    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e97a389b-c8ec-4a4c-b508-443816da8bc1" containerName="main"
Sep 18 19:58:35.761143 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:35.761119    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche2nbmz"
Sep 18 19:58:35.763901 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:35.763878    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Sep 18 19:58:35.764084 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:35.763930    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Sep 18 19:58:35.764084 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:35.764007    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-sa-dockercfg-ssbb7\""
Sep 18 19:58:35.764084 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:35.764038    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-2nc5c\""
Sep 18 19:58:35.764325 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:35.764140    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Sep 18 19:58:35.769476 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:35.769448    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche2nbmz"]
Sep 18 19:58:35.783366 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:35.783279    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:35.838702 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:35.838666    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8kvb\" (UniqueName: \"kubernetes.io/projected/d069c6bb-8208-4b95-a428-ed1b991dec54-kube-api-access-q8kvb\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche2nbmz\" (UID: \"d069c6bb-8208-4b95-a428-ed1b991dec54\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche2nbmz"
Sep 18 19:58:35.838877 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:35.838730    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d069c6bb-8208-4b95-a428-ed1b991dec54-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche2nbmz\" (UID: \"d069c6bb-8208-4b95-a428-ed1b991dec54\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche2nbmz"
Sep 18 19:58:35.939208 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:35.939168    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d069c6bb-8208-4b95-a428-ed1b991dec54-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche2nbmz\" (UID: \"d069c6bb-8208-4b95-a428-ed1b991dec54\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche2nbmz"
Sep 18 19:58:35.939397 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:35.939262    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8kvb\" (UniqueName: \"kubernetes.io/projected/d069c6bb-8208-4b95-a428-ed1b991dec54-kube-api-access-q8kvb\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche2nbmz\" (UID: \"d069c6bb-8208-4b95-a428-ed1b991dec54\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche2nbmz"
Sep 18 19:58:35.941783 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:35.941756    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d069c6bb-8208-4b95-a428-ed1b991dec54-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche2nbmz\" (UID: \"d069c6bb-8208-4b95-a428-ed1b991dec54\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche2nbmz"
Sep 18 19:58:35.946785 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:35.946761    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8kvb\" (UniqueName: \"kubernetes.io/projected/d069c6bb-8208-4b95-a428-ed1b991dec54-kube-api-access-q8kvb\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche2nbmz\" (UID: \"d069c6bb-8208-4b95-a428-ed1b991dec54\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche2nbmz"
Sep 18 19:58:36.073532 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:36.073433    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche2nbmz"
Sep 18 19:58:36.345418 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:36.345323    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche2nbmz"]
Sep 18 19:58:36.348329 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:58:36.348294    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd069c6bb_8208_4b95_a428_ed1b991dec54.slice/crio-024b8c1857fb977ad0d3be90d1ee771b3a2a97dc3b5024e8ee9cedd49ddb21d8 WatchSource:0}: Error finding container 024b8c1857fb977ad0d3be90d1ee771b3a2a97dc3b5024e8ee9cedd49ddb21d8: Status 404 returned error can't find the container with id 024b8c1857fb977ad0d3be90d1ee771b3a2a97dc3b5024e8ee9cedd49ddb21d8
Sep 18 19:58:37.013158 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:37.013123    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche2nbmz" event={"ID":"d069c6bb-8208-4b95-a428-ed1b991dec54","Type":"ContainerStarted","Data":"1b143c7ae7ef883aac32daf96e89694a23a9edabca81413209d2485ca5545fc1"}
Sep 18 19:58:37.013370 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:37.013202    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche2nbmz" event={"ID":"d069c6bb-8208-4b95-a428-ed1b991dec54","Type":"ContainerStarted","Data":"024b8c1857fb977ad0d3be90d1ee771b3a2a97dc3b5024e8ee9cedd49ddb21d8"}
Sep 18 19:58:37.013370 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:37.013247    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche2nbmz"
Sep 18 19:58:37.029310 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:37.029250    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche2nbmz" podStartSLOduration=2.029232992 podStartE2EDuration="2.029232992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:58:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:58:37.026973087 +0000 UTC m=+1233.537329330" watchObservedRunningTime="2026-09-18 19:58:37.029232992 +0000 UTC m=+1233.539589083"
Sep 18 19:58:39.027430 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:39.027395    2564 generic.go:356] "Generic (PLEG): container finished" podID="5fb2a776-c901-46d5-a31c-cbbc1ef1f9b0" containerID="3d266c91651dca660b0aa9987c1e3c84794048fdbb3dd77b80397d6b93cfe208" exitCode=0
Sep 18 19:58:39.027778 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:39.027465    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-wwvlm" event={"ID":"5fb2a776-c901-46d5-a31c-cbbc1ef1f9b0","Type":"ContainerDied","Data":"3d266c91651dca660b0aa9987c1e3c84794048fdbb3dd77b80397d6b93cfe208"}
Sep 18 19:58:39.109445 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:39.109417    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-wwvlm"
Sep 18 19:58:39.164567 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:39.164528    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fb2a776-c901-46d5-a31c-cbbc1ef1f9b0-tls-certs\") pod \"5fb2a776-c901-46d5-a31c-cbbc1ef1f9b0\" (UID: \"5fb2a776-c901-46d5-a31c-cbbc1ef1f9b0\") "
Sep 18 19:58:39.164780 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:39.164608    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zv2n2\" (UniqueName: \"kubernetes.io/projected/5fb2a776-c901-46d5-a31c-cbbc1ef1f9b0-kube-api-access-zv2n2\") pod \"5fb2a776-c901-46d5-a31c-cbbc1ef1f9b0\" (UID: \"5fb2a776-c901-46d5-a31c-cbbc1ef1f9b0\") "
Sep 18 19:58:39.166823 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:39.166785    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fb2a776-c901-46d5-a31c-cbbc1ef1f9b0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5fb2a776-c901-46d5-a31c-cbbc1ef1f9b0" (UID: "5fb2a776-c901-46d5-a31c-cbbc1ef1f9b0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:58:39.166947 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:39.166825    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fb2a776-c901-46d5-a31c-cbbc1ef1f9b0-kube-api-access-zv2n2" (OuterVolumeSpecName: "kube-api-access-zv2n2") pod "5fb2a776-c901-46d5-a31c-cbbc1ef1f9b0" (UID: "5fb2a776-c901-46d5-a31c-cbbc1ef1f9b0"). InnerVolumeSpecName "kube-api-access-zv2n2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:58:39.266091 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:39.266045    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fb2a776-c901-46d5-a31c-cbbc1ef1f9b0-tls-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 19:58:39.266091 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:39.266077    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zv2n2\" (UniqueName: \"kubernetes.io/projected/5fb2a776-c901-46d5-a31c-cbbc1ef1f9b0-kube-api-access-zv2n2\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 19:58:40.002712 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:40.002677    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:40.034351 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:40.034321    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-wwvlm"
Sep 18 19:58:40.034815 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:40.034354    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-wwvlm" event={"ID":"5fb2a776-c901-46d5-a31c-cbbc1ef1f9b0","Type":"ContainerDied","Data":"44e8bfc789b5c0bcfa140f0e57c8a168c0fdf26e014324480d390e31bbb9a24f"}
Sep 18 19:58:40.034815 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:40.034416    2564 scope.go:117] "RemoveContainer" containerID="3d266c91651dca660b0aa9987c1e3c84794048fdbb3dd77b80397d6b93cfe208"
Sep 18 19:58:40.048403 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:40.048374    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-wwvlm"]
Sep 18 19:58:40.051822 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:40.051789    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-wwvlm"]
Sep 18 19:58:41.999627 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:41.999592    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5fb2a776-c901-46d5-a31c-cbbc1ef1f9b0" path="/var/lib/kubelet/pods/5fb2a776-c901-46d5-a31c-cbbc1ef1f9b0/volumes"
Sep 18 19:58:42.772656 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:42.772621    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:42.826169 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:42.826133    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:43.932514 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:43.932475    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:44.031083 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:44.031051    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:50.059852 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:50.059800    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:55.511542 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:55.511502    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:55.626438 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:58:55.626394    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:01.094434 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:59:01.094355    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:01.326441 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:59:01.326407    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ct2j7z"]
Sep 18 19:59:01.326791 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:59:01.326775    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fb2a776-c901-46d5-a31c-cbbc1ef1f9b0" containerName="main"
Sep 18 19:59:01.326791 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:59:01.326791    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fb2a776-c901-46d5-a31c-cbbc1ef1f9b0" containerName="main"
Sep 18 19:59:01.326948 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:59:01.326854    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fb2a776-c901-46d5-a31c-cbbc1ef1f9b0" containerName="main"
Sep 18 19:59:01.331810 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:59:01.331784    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ct2j7z"
Sep 18 19:59:01.334592 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:59:01.334567    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Sep 18 19:59:01.334710 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:59:01.334599    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Sep 18 19:59:01.334710 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:59:01.334570    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Sep 18 19:59:01.334710 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:59:01.334653    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-49p28\""
Sep 18 19:59:01.334891 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:59:01.334788    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-epp-sa-dockercfg-nnvzw\""
Sep 18 19:59:01.338388 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:59:01.338362    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ct2j7z"]
Sep 18 19:59:01.351205 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:59:01.351091    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:01.460937 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:59:01.460899    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjng2\" (UniqueName: \"kubernetes.io/projected/d625c63f-2016-4493-8f1e-824ffdbf37a4-kube-api-access-jjng2\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ct2j7z\" (UID: \"d625c63f-2016-4493-8f1e-824ffdbf37a4\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ct2j7z"
Sep 18 19:59:01.461173 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:59:01.460977    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d625c63f-2016-4493-8f1e-824ffdbf37a4-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ct2j7z\" (UID: \"d625c63f-2016-4493-8f1e-824ffdbf37a4\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ct2j7z"
Sep 18 19:59:01.562416 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:59:01.562375    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d625c63f-2016-4493-8f1e-824ffdbf37a4-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ct2j7z\" (UID: \"d625c63f-2016-4493-8f1e-824ffdbf37a4\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ct2j7z"
Sep 18 19:59:01.562599 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:59:01.562445    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjng2\" (UniqueName: \"kubernetes.io/projected/d625c63f-2016-4493-8f1e-824ffdbf37a4-kube-api-access-jjng2\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ct2j7z\" (UID: \"d625c63f-2016-4493-8f1e-824ffdbf37a4\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ct2j7z"
Sep 18 19:59:01.565199 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:59:01.565169    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d625c63f-2016-4493-8f1e-824ffdbf37a4-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ct2j7z\" (UID: \"d625c63f-2016-4493-8f1e-824ffdbf37a4\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ct2j7z"
Sep 18 19:59:01.570138 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:59:01.570106    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjng2\" (UniqueName: \"kubernetes.io/projected/d625c63f-2016-4493-8f1e-824ffdbf37a4-kube-api-access-jjng2\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ct2j7z\" (UID: \"d625c63f-2016-4493-8f1e-824ffdbf37a4\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ct2j7z"
Sep 18 19:59:01.644518 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:59:01.644484    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ct2j7z"
Sep 18 19:59:01.915655 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:59:01.915623    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ct2j7z"]
Sep 18 19:59:01.918600 ip-10-0-128-161 kubenswrapper[2564]: W0918 19:59:01.918573    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd625c63f_2016_4493_8f1e_824ffdbf37a4.slice/crio-d707a32a142ec51c5e278ba0ea4af89315136da1e13fdba4d1ed172136465456 WatchSource:0}: Error finding container d707a32a142ec51c5e278ba0ea4af89315136da1e13fdba4d1ed172136465456: Status 404 returned error can't find the container with id d707a32a142ec51c5e278ba0ea4af89315136da1e13fdba4d1ed172136465456
Sep 18 19:59:02.141802 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:59:02.141763    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ct2j7z" event={"ID":"d625c63f-2016-4493-8f1e-824ffdbf37a4","Type":"ContainerStarted","Data":"9436c2091c2bc6be330c535746191e3af362d67efaa7ce3ce6429714a36be05c"}
Sep 18 19:59:02.142311 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:59:02.141822    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ct2j7z" event={"ID":"d625c63f-2016-4493-8f1e-824ffdbf37a4","Type":"ContainerStarted","Data":"d707a32a142ec51c5e278ba0ea4af89315136da1e13fdba4d1ed172136465456"}
Sep 18 19:59:02.142311 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:59:02.141843    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ct2j7z"
Sep 18 19:59:02.161833 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:59:02.161776    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ct2j7z" podStartSLOduration=1.161758041 podStartE2EDuration="1.161758041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:59:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:59:02.158445799 +0000 UTC m=+1258.668801897" watchObservedRunningTime="2026-09-18 19:59:02.161758041 +0000 UTC m=+1258.672114131"
Sep 18 19:59:08.020158 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:59:08.020120    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche2nbmz"
Sep 18 19:59:11.569533 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:59:11.569497    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:14.834002 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:59:14.833952    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:14.834717 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:59:14.834368    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche2nbmz"]
Sep 18 19:59:14.927840 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:59:14.927802    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:18.020209 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:59:18.020166    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche2nbmz" podUID="d069c6bb-8208-4b95-a428-ed1b991dec54" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:59:28.019846 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:59:28.019812    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche2nbmz" podUID="d069c6bb-8208-4b95-a428-ed1b991dec54" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:59:29.860953 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:59:29.860907    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche2nbmz" podUID="d069c6bb-8208-4b95-a428-ed1b991dec54" containerName="main" containerID="cri-o://1b143c7ae7ef883aac32daf96e89694a23a9edabca81413209d2485ca5545fc1" gracePeriod=45
Sep 18 19:59:33.148357 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:59:33.148323    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ct2j7z"
Sep 18 19:59:38.020440 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:59:38.020400    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche2nbmz" podUID="d069c6bb-8208-4b95-a428-ed1b991dec54" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:59:38.020826 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:59:38.020516    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche2nbmz"
Sep 18 19:59:48.020478 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:59:48.020439    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche2nbmz" podUID="d069c6bb-8208-4b95-a428-ed1b991dec54" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:59:58.019777 ip-10-0-128-161 kubenswrapper[2564]: I0918 19:59:58.019739    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche2nbmz" podUID="d069c6bb-8208-4b95-a428-ed1b991dec54" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:00:00.117790 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:00:00.117758    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche2nbmz"
Sep 18 20:00:00.291823 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:00:00.291779    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q8kvb\" (UniqueName: \"kubernetes.io/projected/d069c6bb-8208-4b95-a428-ed1b991dec54-kube-api-access-q8kvb\") pod \"d069c6bb-8208-4b95-a428-ed1b991dec54\" (UID: \"d069c6bb-8208-4b95-a428-ed1b991dec54\") "
Sep 18 20:00:00.292027 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:00:00.291875    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d069c6bb-8208-4b95-a428-ed1b991dec54-tls-certs\") pod \"d069c6bb-8208-4b95-a428-ed1b991dec54\" (UID: \"d069c6bb-8208-4b95-a428-ed1b991dec54\") "
Sep 18 20:00:00.294200 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:00:00.294166    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d069c6bb-8208-4b95-a428-ed1b991dec54-kube-api-access-q8kvb" (OuterVolumeSpecName: "kube-api-access-q8kvb") pod "d069c6bb-8208-4b95-a428-ed1b991dec54" (UID: "d069c6bb-8208-4b95-a428-ed1b991dec54"). InnerVolumeSpecName "kube-api-access-q8kvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:00:00.294337 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:00:00.294246    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d069c6bb-8208-4b95-a428-ed1b991dec54-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d069c6bb-8208-4b95-a428-ed1b991dec54" (UID: "d069c6bb-8208-4b95-a428-ed1b991dec54"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:00:00.393369 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:00:00.393311    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d069c6bb-8208-4b95-a428-ed1b991dec54-tls-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:00:00.393369 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:00:00.393365    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8kvb\" (UniqueName: \"kubernetes.io/projected/d069c6bb-8208-4b95-a428-ed1b991dec54-kube-api-access-q8kvb\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:00:00.441609 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:00:00.441575    2564 generic.go:356] "Generic (PLEG): container finished" podID="d069c6bb-8208-4b95-a428-ed1b991dec54" containerID="1b143c7ae7ef883aac32daf96e89694a23a9edabca81413209d2485ca5545fc1" exitCode=0
Sep 18 20:00:00.441780 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:00:00.441616    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche2nbmz" event={"ID":"d069c6bb-8208-4b95-a428-ed1b991dec54","Type":"ContainerDied","Data":"1b143c7ae7ef883aac32daf96e89694a23a9edabca81413209d2485ca5545fc1"}
Sep 18 20:00:00.441780 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:00:00.441622    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche2nbmz"
Sep 18 20:00:00.441780 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:00:00.441658    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche2nbmz" event={"ID":"d069c6bb-8208-4b95-a428-ed1b991dec54","Type":"ContainerDied","Data":"024b8c1857fb977ad0d3be90d1ee771b3a2a97dc3b5024e8ee9cedd49ddb21d8"}
Sep 18 20:00:00.441780 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:00:00.441680    2564 scope.go:117] "RemoveContainer" containerID="1b143c7ae7ef883aac32daf96e89694a23a9edabca81413209d2485ca5545fc1"
Sep 18 20:00:00.450963 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:00:00.450937    2564 scope.go:117] "RemoveContainer" containerID="1b143c7ae7ef883aac32daf96e89694a23a9edabca81413209d2485ca5545fc1"
Sep 18 20:00:00.451289 ip-10-0-128-161 kubenswrapper[2564]: E0918 20:00:00.451269    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b143c7ae7ef883aac32daf96e89694a23a9edabca81413209d2485ca5545fc1\": container with ID starting with 1b143c7ae7ef883aac32daf96e89694a23a9edabca81413209d2485ca5545fc1 not found: ID does not exist" containerID="1b143c7ae7ef883aac32daf96e89694a23a9edabca81413209d2485ca5545fc1"
Sep 18 20:00:00.451368 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:00:00.451296    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b143c7ae7ef883aac32daf96e89694a23a9edabca81413209d2485ca5545fc1"} err="failed to get container status \"1b143c7ae7ef883aac32daf96e89694a23a9edabca81413209d2485ca5545fc1\": rpc error: code = NotFound desc = could not find container \"1b143c7ae7ef883aac32daf96e89694a23a9edabca81413209d2485ca5545fc1\": container with ID starting with 1b143c7ae7ef883aac32daf96e89694a23a9edabca81413209d2485ca5545fc1 not found: ID does not exist"
Sep 18 20:00:00.463638 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:00:00.463604    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche2nbmz"]
Sep 18 20:00:00.469726 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:00:00.469701    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche2nbmz"]
Sep 18 20:00:02.000081 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:00:01.999967    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d069c6bb-8208-4b95-a428-ed1b991dec54" path="/var/lib/kubelet/pods/d069c6bb-8208-4b95-a428-ed1b991dec54/volumes"
Sep 18 20:00:04.603679 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:00:04.603635    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:00:06.485047 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:00:06.485003    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:00:10.366807 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:00:10.366744    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:00:16.085494 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:00:16.085450    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:00:16.185940 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:00:16.185896    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:00:16.532498 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:00:16.532463    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:00:16.767937 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:00:16.767902    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9p78fw"]
Sep 18 20:00:16.768348 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:00:16.768331    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d069c6bb-8208-4b95-a428-ed1b991dec54" containerName="main"
Sep 18 20:00:16.768348 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:00:16.768350    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d069c6bb-8208-4b95-a428-ed1b991dec54" containerName="main"
Sep 18 20:00:16.768458 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:00:16.768418    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d069c6bb-8208-4b95-a428-ed1b991dec54" containerName="main"
Sep 18 20:00:16.772448 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:00:16.772427    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9p78fw"
Sep 18 20:00:16.775357 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:00:16.775336    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-n42ll\""
Sep 18 20:00:16.775357 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:00:16.775346    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Sep 18 20:00:16.775556 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:00:16.775336    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-epp-sa-dockercfg-dkjvt\""
Sep 18 20:00:16.775556 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:00:16.775338    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Sep 18 20:00:16.775556 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:00:16.775336    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Sep 18 20:00:16.778424 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:00:16.778400    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9p78fw"]
Sep 18 20:00:16.792121 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:00:16.792048    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:00:16.837235 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:00:16.837199    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/198b1ad6-a89d-4cf3-8f43-f2615d60db80-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9p78fw\" (UID: \"198b1ad6-a89d-4cf3-8f43-f2615d60db80\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9p78fw"
Sep 18 20:00:16.837416 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:00:16.837245    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rddrc\" (UniqueName: \"kubernetes.io/projected/198b1ad6-a89d-4cf3-8f43-f2615d60db80-kube-api-access-rddrc\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9p78fw\" (UID: \"198b1ad6-a89d-4cf3-8f43-f2615d60db80\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9p78fw"
Sep 18 20:00:16.937871 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:00:16.937832    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/198b1ad6-a89d-4cf3-8f43-f2615d60db80-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9p78fw\" (UID: \"198b1ad6-a89d-4cf3-8f43-f2615d60db80\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9p78fw"
Sep 18 20:00:16.938104 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:00:16.937879    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rddrc\" (UniqueName: \"kubernetes.io/projected/198b1ad6-a89d-4cf3-8f43-f2615d60db80-kube-api-access-rddrc\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9p78fw\" (UID: \"198b1ad6-a89d-4cf3-8f43-f2615d60db80\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9p78fw"
Sep 18 20:00:16.940540 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:00:16.940510    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/198b1ad6-a89d-4cf3-8f43-f2615d60db80-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9p78fw\" (UID: \"198b1ad6-a89d-4cf3-8f43-f2615d60db80\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9p78fw"
Sep 18 20:00:16.945380 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:00:16.945335    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rddrc\" (UniqueName: \"kubernetes.io/projected/198b1ad6-a89d-4cf3-8f43-f2615d60db80-kube-api-access-rddrc\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9p78fw\" (UID: \"198b1ad6-a89d-4cf3-8f43-f2615d60db80\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9p78fw"
Sep 18 20:00:17.084025 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:00:17.083900    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9p78fw"
Sep 18 20:00:17.355664 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:00:17.355571    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9p78fw"]
Sep 18 20:00:17.359047 ip-10-0-128-161 kubenswrapper[2564]: W0918 20:00:17.359013    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod198b1ad6_a89d_4cf3_8f43_f2615d60db80.slice/crio-3ac9e541fba5a6e4c1cc3332fa5e934c68c0e982be5e3d71e0e21f97f4077343 WatchSource:0}: Error finding container 3ac9e541fba5a6e4c1cc3332fa5e934c68c0e982be5e3d71e0e21f97f4077343: Status 404 returned error can't find the container with id 3ac9e541fba5a6e4c1cc3332fa5e934c68c0e982be5e3d71e0e21f97f4077343
Sep 18 20:00:17.529746 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:00:17.529711    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9p78fw" event={"ID":"198b1ad6-a89d-4cf3-8f43-f2615d60db80","Type":"ContainerStarted","Data":"87f6b3d29b69f337d826d2329ec4fd32092c03e8c2e0980ca167cf8e9ad0a1ed"}
Sep 18 20:00:17.529958 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:00:17.529767    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9p78fw" event={"ID":"198b1ad6-a89d-4cf3-8f43-f2615d60db80","Type":"ContainerStarted","Data":"3ac9e541fba5a6e4c1cc3332fa5e934c68c0e982be5e3d71e0e21f97f4077343"}
Sep 18 20:00:17.529958 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:00:17.529807    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9p78fw"
Sep 18 20:00:17.545639 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:00:17.545580    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9p78fw" podStartSLOduration=1.545558589 podStartE2EDuration="1.545558589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:00:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:00:17.545291533 +0000 UTC m=+1334.055647662" watchObservedRunningTime="2026-09-18 20:00:17.545558589 +0000 UTC m=+1334.055914680"
Sep 18 20:00:33.842706 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:00:33.842660    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:00:48.536321 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:00:48.536292    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9p78fw"
Sep 18 20:00:49.932751 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:00:49.932709    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9p78fw"]
Sep 18 20:00:49.940728 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:00:49.940687    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:00:50.029814 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:00:50.029758    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:00:58.536422 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:00:58.536381    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9p78fw" podUID="198b1ad6-a89d-4cf3-8f43-f2615d60db80" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:01:03.211258 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:03.211207    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ct2j7z"]
Sep 18 20:01:03.224590 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:03.224544    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:01:03.323369 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:03.323330    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:01:04.958695 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:04.958656    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9p78fw" podUID="198b1ad6-a89d-4cf3-8f43-f2615d60db80" containerName="main" containerID="cri-o://87f6b3d29b69f337d826d2329ec4fd32092c03e8c2e0980ca167cf8e9ad0a1ed" gracePeriod=45
Sep 18 20:01:08.536375 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:08.536332    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9p78fw" podUID="198b1ad6-a89d-4cf3-8f43-f2615d60db80" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:01:13.148207 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:13.148169    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ct2j7z" podUID="d625c63f-2016-4493-8f1e-824ffdbf37a4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:01:18.235012 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:18.234933    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ct2j7z" podUID="d625c63f-2016-4493-8f1e-824ffdbf37a4" containerName="main" containerID="cri-o://9436c2091c2bc6be330c535746191e3af362d67efaa7ce3ce6429714a36be05c" gracePeriod=45
Sep 18 20:01:18.535882 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:18.535797    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9p78fw" podUID="198b1ad6-a89d-4cf3-8f43-f2615d60db80" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:01:18.536072 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:18.535897    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9p78fw"
Sep 18 20:01:23.148998 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:23.148953    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ct2j7z" podUID="d625c63f-2016-4493-8f1e-824ffdbf37a4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:01:28.536568 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:28.536522    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9p78fw" podUID="198b1ad6-a89d-4cf3-8f43-f2615d60db80" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:01:33.149070 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:33.149028    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ct2j7z" podUID="d625c63f-2016-4493-8f1e-824ffdbf37a4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:01:33.149555 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:33.149176    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ct2j7z"
Sep 18 20:01:35.202486 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:35.202457    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9p78fw"
Sep 18 20:01:35.335477 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:35.335381    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rddrc\" (UniqueName: \"kubernetes.io/projected/198b1ad6-a89d-4cf3-8f43-f2615d60db80-kube-api-access-rddrc\") pod \"198b1ad6-a89d-4cf3-8f43-f2615d60db80\" (UID: \"198b1ad6-a89d-4cf3-8f43-f2615d60db80\") "
Sep 18 20:01:35.335644 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:35.335501    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/198b1ad6-a89d-4cf3-8f43-f2615d60db80-tls-certs\") pod \"198b1ad6-a89d-4cf3-8f43-f2615d60db80\" (UID: \"198b1ad6-a89d-4cf3-8f43-f2615d60db80\") "
Sep 18 20:01:35.337759 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:35.337729    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/198b1ad6-a89d-4cf3-8f43-f2615d60db80-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "198b1ad6-a89d-4cf3-8f43-f2615d60db80" (UID: "198b1ad6-a89d-4cf3-8f43-f2615d60db80"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:01:35.337861 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:35.337754    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/198b1ad6-a89d-4cf3-8f43-f2615d60db80-kube-api-access-rddrc" (OuterVolumeSpecName: "kube-api-access-rddrc") pod "198b1ad6-a89d-4cf3-8f43-f2615d60db80" (UID: "198b1ad6-a89d-4cf3-8f43-f2615d60db80"). InnerVolumeSpecName "kube-api-access-rddrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:01:35.436780 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:35.436734    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/198b1ad6-a89d-4cf3-8f43-f2615d60db80-tls-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:01:35.436780 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:35.436769    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rddrc\" (UniqueName: \"kubernetes.io/projected/198b1ad6-a89d-4cf3-8f43-f2615d60db80-kube-api-access-rddrc\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:01:35.922634 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:35.922601    2564 generic.go:356] "Generic (PLEG): container finished" podID="198b1ad6-a89d-4cf3-8f43-f2615d60db80" containerID="87f6b3d29b69f337d826d2329ec4fd32092c03e8c2e0980ca167cf8e9ad0a1ed" exitCode=0
Sep 18 20:01:35.922856 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:35.922650    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9p78fw"
Sep 18 20:01:35.922856 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:35.922689    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9p78fw" event={"ID":"198b1ad6-a89d-4cf3-8f43-f2615d60db80","Type":"ContainerDied","Data":"87f6b3d29b69f337d826d2329ec4fd32092c03e8c2e0980ca167cf8e9ad0a1ed"}
Sep 18 20:01:35.922856 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:35.922751    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9p78fw" event={"ID":"198b1ad6-a89d-4cf3-8f43-f2615d60db80","Type":"ContainerDied","Data":"3ac9e541fba5a6e4c1cc3332fa5e934c68c0e982be5e3d71e0e21f97f4077343"}
Sep 18 20:01:35.922856 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:35.922783    2564 scope.go:117] "RemoveContainer" containerID="87f6b3d29b69f337d826d2329ec4fd32092c03e8c2e0980ca167cf8e9ad0a1ed"
Sep 18 20:01:35.934975 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:35.934953    2564 scope.go:117] "RemoveContainer" containerID="87f6b3d29b69f337d826d2329ec4fd32092c03e8c2e0980ca167cf8e9ad0a1ed"
Sep 18 20:01:35.935346 ip-10-0-128-161 kubenswrapper[2564]: E0918 20:01:35.935324    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87f6b3d29b69f337d826d2329ec4fd32092c03e8c2e0980ca167cf8e9ad0a1ed\": container with ID starting with 87f6b3d29b69f337d826d2329ec4fd32092c03e8c2e0980ca167cf8e9ad0a1ed not found: ID does not exist" containerID="87f6b3d29b69f337d826d2329ec4fd32092c03e8c2e0980ca167cf8e9ad0a1ed"
Sep 18 20:01:35.935440 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:35.935356    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87f6b3d29b69f337d826d2329ec4fd32092c03e8c2e0980ca167cf8e9ad0a1ed"} err="failed to get container status \"87f6b3d29b69f337d826d2329ec4fd32092c03e8c2e0980ca167cf8e9ad0a1ed\": rpc error: code = NotFound desc = could not find container \"87f6b3d29b69f337d826d2329ec4fd32092c03e8c2e0980ca167cf8e9ad0a1ed\": container with ID starting with 87f6b3d29b69f337d826d2329ec4fd32092c03e8c2e0980ca167cf8e9ad0a1ed not found: ID does not exist"
Sep 18 20:01:35.945681 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:35.945653    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9p78fw"]
Sep 18 20:01:35.951739 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:35.951708    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9p78fw"]
Sep 18 20:01:35.999889 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:35.999858    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="198b1ad6-a89d-4cf3-8f43-f2615d60db80" path="/var/lib/kubelet/pods/198b1ad6-a89d-4cf3-8f43-f2615d60db80/volumes"
Sep 18 20:01:39.602932 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:39.602823    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:01:39.703644 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:39.703604    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:01:43.148293 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:43.148246    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ct2j7z" podUID="d625c63f-2016-4493-8f1e-824ffdbf37a4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:01:45.632820 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:45.632783    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:01:48.500263 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:48.500199    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ct2j7z"
Sep 18 20:01:48.652712 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:48.652665    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jjng2\" (UniqueName: \"kubernetes.io/projected/d625c63f-2016-4493-8f1e-824ffdbf37a4-kube-api-access-jjng2\") pod \"d625c63f-2016-4493-8f1e-824ffdbf37a4\" (UID: \"d625c63f-2016-4493-8f1e-824ffdbf37a4\") "
Sep 18 20:01:48.652712 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:48.652709    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d625c63f-2016-4493-8f1e-824ffdbf37a4-tls-certs\") pod \"d625c63f-2016-4493-8f1e-824ffdbf37a4\" (UID: \"d625c63f-2016-4493-8f1e-824ffdbf37a4\") "
Sep 18 20:01:48.655062 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:48.655034    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d625c63f-2016-4493-8f1e-824ffdbf37a4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d625c63f-2016-4493-8f1e-824ffdbf37a4" (UID: "d625c63f-2016-4493-8f1e-824ffdbf37a4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:01:48.655062 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:48.655049    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d625c63f-2016-4493-8f1e-824ffdbf37a4-kube-api-access-jjng2" (OuterVolumeSpecName: "kube-api-access-jjng2") pod "d625c63f-2016-4493-8f1e-824ffdbf37a4" (UID: "d625c63f-2016-4493-8f1e-824ffdbf37a4"). InnerVolumeSpecName "kube-api-access-jjng2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:01:48.754378 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:48.754275    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjng2\" (UniqueName: \"kubernetes.io/projected/d625c63f-2016-4493-8f1e-824ffdbf37a4-kube-api-access-jjng2\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:01:48.754378 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:48.754309    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d625c63f-2016-4493-8f1e-824ffdbf37a4-tls-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:01:48.987777 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:48.987734    2564 generic.go:356] "Generic (PLEG): container finished" podID="d625c63f-2016-4493-8f1e-824ffdbf37a4" containerID="9436c2091c2bc6be330c535746191e3af362d67efaa7ce3ce6429714a36be05c" exitCode=0
Sep 18 20:01:48.987777 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:48.987784    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ct2j7z"
Sep 18 20:01:48.988025 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:48.987819    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ct2j7z" event={"ID":"d625c63f-2016-4493-8f1e-824ffdbf37a4","Type":"ContainerDied","Data":"9436c2091c2bc6be330c535746191e3af362d67efaa7ce3ce6429714a36be05c"}
Sep 18 20:01:48.988025 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:48.987878    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ct2j7z" event={"ID":"d625c63f-2016-4493-8f1e-824ffdbf37a4","Type":"ContainerDied","Data":"d707a32a142ec51c5e278ba0ea4af89315136da1e13fdba4d1ed172136465456"}
Sep 18 20:01:48.988025 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:48.987900    2564 scope.go:117] "RemoveContainer" containerID="9436c2091c2bc6be330c535746191e3af362d67efaa7ce3ce6429714a36be05c"
Sep 18 20:01:49.002922 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:49.002885    2564 scope.go:117] "RemoveContainer" containerID="9436c2091c2bc6be330c535746191e3af362d67efaa7ce3ce6429714a36be05c"
Sep 18 20:01:49.003403 ip-10-0-128-161 kubenswrapper[2564]: E0918 20:01:49.003366    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9436c2091c2bc6be330c535746191e3af362d67efaa7ce3ce6429714a36be05c\": container with ID starting with 9436c2091c2bc6be330c535746191e3af362d67efaa7ce3ce6429714a36be05c not found: ID does not exist" containerID="9436c2091c2bc6be330c535746191e3af362d67efaa7ce3ce6429714a36be05c"
Sep 18 20:01:49.003463 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:49.003422    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9436c2091c2bc6be330c535746191e3af362d67efaa7ce3ce6429714a36be05c"} err="failed to get container status \"9436c2091c2bc6be330c535746191e3af362d67efaa7ce3ce6429714a36be05c\": rpc error: code = NotFound desc = could not find container \"9436c2091c2bc6be330c535746191e3af362d67efaa7ce3ce6429714a36be05c\": container with ID starting with 9436c2091c2bc6be330c535746191e3af362d67efaa7ce3ce6429714a36be05c not found: ID does not exist"
Sep 18 20:01:49.010935 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:49.010866    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ct2j7z"]
Sep 18 20:01:49.014658 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:49.014632    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ct2j7z"]
Sep 18 20:01:49.999422 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:49.999387    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d625c63f-2016-4493-8f1e-824ffdbf37a4" path="/var/lib/kubelet/pods/d625c63f-2016-4493-8f1e-824ffdbf37a4/volumes"
Sep 18 20:01:51.078423 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:51.078388    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:01:51.178227 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:51.178178    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:01:52.729153 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:52.729117    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:01:52.925889 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:52.925854    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5t9j29"]
Sep 18 20:01:52.926284 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:52.926269    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d625c63f-2016-4493-8f1e-824ffdbf37a4" containerName="main"
Sep 18 20:01:52.926343 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:52.926286    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d625c63f-2016-4493-8f1e-824ffdbf37a4" containerName="main"
Sep 18 20:01:52.926343 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:52.926301    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="198b1ad6-a89d-4cf3-8f43-f2615d60db80" containerName="main"
Sep 18 20:01:52.926343 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:52.926306    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="198b1ad6-a89d-4cf3-8f43-f2615d60db80" containerName="main"
Sep 18 20:01:52.926455 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:52.926373    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d625c63f-2016-4493-8f1e-824ffdbf37a4" containerName="main"
Sep 18 20:01:52.926455 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:52.926383    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="198b1ad6-a89d-4cf3-8f43-f2615d60db80" containerName="main"
Sep 18 20:01:52.931229 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:52.931202    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5t9j29"
Sep 18 20:01:52.934000 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:52.933931    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 18 20:01:52.934213 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:52.934099    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-jh2w2\""
Sep 18 20:01:52.934306 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:52.934212    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 18 20:01:52.934364 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:52.934317    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-pwlm2\""
Sep 18 20:01:52.934399 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:52.934370    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 18 20:01:52.939695 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:52.939666    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5t9j29"]
Sep 18 20:01:52.966041 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:52.966009    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:01:52.992330 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:52.992220    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb87e91c-c74f-4264-abe2-4831658c045b-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5t9j29\" (UID: \"cb87e91c-c74f-4264-abe2-4831658c045b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5t9j29"
Sep 18 20:01:52.992330 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:52.992276    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf28t\" (UniqueName: \"kubernetes.io/projected/cb87e91c-c74f-4264-abe2-4831658c045b-kube-api-access-zf28t\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5t9j29\" (UID: \"cb87e91c-c74f-4264-abe2-4831658c045b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5t9j29"
Sep 18 20:01:53.092908 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:53.092868    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb87e91c-c74f-4264-abe2-4831658c045b-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5t9j29\" (UID: \"cb87e91c-c74f-4264-abe2-4831658c045b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5t9j29"
Sep 18 20:01:53.092908 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:53.092915    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf28t\" (UniqueName: \"kubernetes.io/projected/cb87e91c-c74f-4264-abe2-4831658c045b-kube-api-access-zf28t\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5t9j29\" (UID: \"cb87e91c-c74f-4264-abe2-4831658c045b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5t9j29"
Sep 18 20:01:53.095537 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:53.095510    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb87e91c-c74f-4264-abe2-4831658c045b-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5t9j29\" (UID: \"cb87e91c-c74f-4264-abe2-4831658c045b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5t9j29"
Sep 18 20:01:53.100600 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:53.100568    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf28t\" (UniqueName: \"kubernetes.io/projected/cb87e91c-c74f-4264-abe2-4831658c045b-kube-api-access-zf28t\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5t9j29\" (UID: \"cb87e91c-c74f-4264-abe2-4831658c045b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5t9j29"
Sep 18 20:01:53.176876 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:53.176835    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:01:53.246973 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:53.246883    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5t9j29"
Sep 18 20:01:53.272071 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:53.271976    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:01:53.521632 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:53.521409    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5t9j29"]
Sep 18 20:01:53.524609 ip-10-0-128-161 kubenswrapper[2564]: W0918 20:01:53.524576    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb87e91c_c74f_4264_abe2_4831658c045b.slice/crio-4ed35654fd55e49934ebb24f68aa04905b0d84b40d1e47ae612a5e997dd915dd WatchSource:0}: Error finding container 4ed35654fd55e49934ebb24f68aa04905b0d84b40d1e47ae612a5e997dd915dd: Status 404 returned error can't find the container with id 4ed35654fd55e49934ebb24f68aa04905b0d84b40d1e47ae612a5e997dd915dd
Sep 18 20:01:54.016216 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:54.016183    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5t9j29" event={"ID":"cb87e91c-c74f-4264-abe2-4831658c045b","Type":"ContainerStarted","Data":"f21405aa39c19bc9094da49c4dad6fd24354556cc9d4aa70b1d2126608e8c15e"}
Sep 18 20:01:54.016718 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:54.016238    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5t9j29" event={"ID":"cb87e91c-c74f-4264-abe2-4831658c045b","Type":"ContainerStarted","Data":"4ed35654fd55e49934ebb24f68aa04905b0d84b40d1e47ae612a5e997dd915dd"}
Sep 18 20:01:54.016718 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:54.016273    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5t9j29"
Sep 18 20:01:54.032475 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:54.032419    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5t9j29" podStartSLOduration=2.032401271 podStartE2EDuration="2.032401271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:01:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:01:54.030322853 +0000 UTC m=+1430.540678952" watchObservedRunningTime="2026-09-18 20:01:54.032401271 +0000 UTC m=+1430.542757361"
Sep 18 20:01:59.154351 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:01:59.154266    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:02:00.306700 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:02:00.306663    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:02:04.880906 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:02:04.880860    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:02:04.943119 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:02:04.943070    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:02:10.930361 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:02:10.930326    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:02:11.152651 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:02:11.152610    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-8lvwj"]
Sep 18 20:02:11.157376 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:02:11.157350    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-8lvwj"
Sep 18 20:02:11.160281 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:02:11.160251    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Sep 18 20:02:11.160281 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:02:11.160287    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-4hsxs\""
Sep 18 20:02:11.160534 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:02:11.160252    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Sep 18 20:02:11.160534 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:02:11.160376    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-epp-sa-dockercfg-qrdrl\""
Sep 18 20:02:11.160646 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:02:11.160629    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Sep 18 20:02:11.165894 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:02:11.165628    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-8lvwj"]
Sep 18 20:02:11.180183 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:02:11.180146    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:02:11.255698 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:02:11.255608    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0578992-b41a-4605-b3a4-8d25c215f74b-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-8lvwj\" (UID: \"a0578992-b41a-4605-b3a4-8d25c215f74b\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-8lvwj"
Sep 18 20:02:11.255853 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:02:11.255697    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ndqs\" (UniqueName: \"kubernetes.io/projected/a0578992-b41a-4605-b3a4-8d25c215f74b-kube-api-access-7ndqs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-8lvwj\" (UID: \"a0578992-b41a-4605-b3a4-8d25c215f74b\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-8lvwj"
Sep 18 20:02:11.356587 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:02:11.356540    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ndqs\" (UniqueName: \"kubernetes.io/projected/a0578992-b41a-4605-b3a4-8d25c215f74b-kube-api-access-7ndqs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-8lvwj\" (UID: \"a0578992-b41a-4605-b3a4-8d25c215f74b\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-8lvwj"
Sep 18 20:02:11.356783 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:02:11.356701    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0578992-b41a-4605-b3a4-8d25c215f74b-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-8lvwj\" (UID: \"a0578992-b41a-4605-b3a4-8d25c215f74b\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-8lvwj"
Sep 18 20:02:11.359549 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:02:11.359521    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0578992-b41a-4605-b3a4-8d25c215f74b-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-8lvwj\" (UID: \"a0578992-b41a-4605-b3a4-8d25c215f74b\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-8lvwj"
Sep 18 20:02:11.364816 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:02:11.364785    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ndqs\" (UniqueName: \"kubernetes.io/projected/a0578992-b41a-4605-b3a4-8d25c215f74b-kube-api-access-7ndqs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-8lvwj\" (UID: \"a0578992-b41a-4605-b3a4-8d25c215f74b\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-8lvwj"
Sep 18 20:02:11.470644 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:02:11.470599    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-8lvwj"
Sep 18 20:02:11.742883 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:02:11.742845    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-8lvwj"]
Sep 18 20:02:11.747340 ip-10-0-128-161 kubenswrapper[2564]: W0918 20:02:11.747307    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0578992_b41a_4605_b3a4_8d25c215f74b.slice/crio-a2e53f9abfef8f69931026c27e2acc624b23b509a6ab44a93381c4fb1a70cd2f WatchSource:0}: Error finding container a2e53f9abfef8f69931026c27e2acc624b23b509a6ab44a93381c4fb1a70cd2f: Status 404 returned error can't find the container with id a2e53f9abfef8f69931026c27e2acc624b23b509a6ab44a93381c4fb1a70cd2f
Sep 18 20:02:12.106473 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:02:12.106368    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-8lvwj" event={"ID":"a0578992-b41a-4605-b3a4-8d25c215f74b","Type":"ContainerStarted","Data":"09965a64e9cb49722d1e647208c853ef83a0acade4764bf563f503bd6c531d3d"}
Sep 18 20:02:12.106473 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:02:12.106421    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-8lvwj" event={"ID":"a0578992-b41a-4605-b3a4-8d25c215f74b","Type":"ContainerStarted","Data":"a2e53f9abfef8f69931026c27e2acc624b23b509a6ab44a93381c4fb1a70cd2f"}
Sep 18 20:02:12.106473 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:02:12.106453    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-8lvwj"
Sep 18 20:02:12.125498 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:02:12.125445    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-8lvwj" podStartSLOduration=1.125425635 podStartE2EDuration="1.125425635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:02:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:02:12.123679816 +0000 UTC m=+1448.634035923" watchObservedRunningTime="2026-09-18 20:02:12.125425635 +0000 UTC m=+1448.635781767"
Sep 18 20:02:25.022565 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:02:25.022535    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5t9j29"
Sep 18 20:02:25.041681 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:02:25.041641    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:02:28.539167 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:02:28.538723    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:02:28.549353 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:02:28.549289    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5t9j29"]
Sep 18 20:02:28.639427 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:02:28.639387    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:02:35.024132 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:02:35.024092    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5t9j29" podUID="cb87e91c-c74f-4264-abe2-4831658c045b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:02:39.878664 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:02:39.878622    2564 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5t9j29" message="" err="command '/bin/sleep 15' exited with 137: "
Sep 18 20:02:39.878970 ip-10-0-128-161 kubenswrapper[2564]: E0918 20:02:39.878669    2564 kuberuntime_container.go:755] "PreStop hook failed" err="command '/bin/sleep 15' exited with 137: " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5t9j29" podUID="cb87e91c-c74f-4264-abe2-4831658c045b" containerName="main" containerID="cri-o://f21405aa39c19bc9094da49c4dad6fd24354556cc9d4aa70b1d2126608e8c15e"
Sep 18 20:02:39.878970 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:02:39.878735    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5t9j29" podUID="cb87e91c-c74f-4264-abe2-4831658c045b" containerName="main" containerID="cri-o://f21405aa39c19bc9094da49c4dad6fd24354556cc9d4aa70b1d2126608e8c15e" gracePeriod=49
Sep 18 20:02:40.049052 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:02:40.049025    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5t9j29_cb87e91c-c74f-4264-abe2-4831658c045b/main/0.log"
Sep 18 20:02:40.051413 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:02:40.051395    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5t9j29"
Sep 18 20:02:40.216932 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:02:40.216893    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zf28t\" (UniqueName: \"kubernetes.io/projected/cb87e91c-c74f-4264-abe2-4831658c045b-kube-api-access-zf28t\") pod \"cb87e91c-c74f-4264-abe2-4831658c045b\" (UID: \"cb87e91c-c74f-4264-abe2-4831658c045b\") "
Sep 18 20:02:40.217149 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:02:40.216947    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb87e91c-c74f-4264-abe2-4831658c045b-tls-certs\") pod \"cb87e91c-c74f-4264-abe2-4831658c045b\" (UID: \"cb87e91c-c74f-4264-abe2-4831658c045b\") "
Sep 18 20:02:40.219432 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:02:40.219394    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb87e91c-c74f-4264-abe2-4831658c045b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cb87e91c-c74f-4264-abe2-4831658c045b" (UID: "cb87e91c-c74f-4264-abe2-4831658c045b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:02:40.219579 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:02:40.219453    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb87e91c-c74f-4264-abe2-4831658c045b-kube-api-access-zf28t" (OuterVolumeSpecName: "kube-api-access-zf28t") pod "cb87e91c-c74f-4264-abe2-4831658c045b" (UID: "cb87e91c-c74f-4264-abe2-4831658c045b"). InnerVolumeSpecName "kube-api-access-zf28t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:02:40.242346 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:02:40.242318    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5t9j29_cb87e91c-c74f-4264-abe2-4831658c045b/main/0.log"
Sep 18 20:02:40.242548 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:02:40.242362    2564 generic.go:356] "Generic (PLEG): container finished" podID="cb87e91c-c74f-4264-abe2-4831658c045b" containerID="f21405aa39c19bc9094da49c4dad6fd24354556cc9d4aa70b1d2126608e8c15e" exitCode=1
Sep 18 20:02:40.242548 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:02:40.242395    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5t9j29" event={"ID":"cb87e91c-c74f-4264-abe2-4831658c045b","Type":"ContainerDied","Data":"f21405aa39c19bc9094da49c4dad6fd24354556cc9d4aa70b1d2126608e8c15e"}
Sep 18 20:02:40.242548 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:02:40.242427    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5t9j29"
Sep 18 20:02:40.242548 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:02:40.242444    2564 scope.go:117] "RemoveContainer" containerID="f21405aa39c19bc9094da49c4dad6fd24354556cc9d4aa70b1d2126608e8c15e"
Sep 18 20:02:40.242548 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:02:40.242434    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5t9j29" event={"ID":"cb87e91c-c74f-4264-abe2-4831658c045b","Type":"ContainerDied","Data":"4ed35654fd55e49934ebb24f68aa04905b0d84b40d1e47ae612a5e997dd915dd"}
Sep 18 20:02:40.251849 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:02:40.251824    2564 scope.go:117] "RemoveContainer" containerID="f21405aa39c19bc9094da49c4dad6fd24354556cc9d4aa70b1d2126608e8c15e"
Sep 18 20:02:40.252204 ip-10-0-128-161 kubenswrapper[2564]: E0918 20:02:40.252180    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f21405aa39c19bc9094da49c4dad6fd24354556cc9d4aa70b1d2126608e8c15e\": container with ID starting with f21405aa39c19bc9094da49c4dad6fd24354556cc9d4aa70b1d2126608e8c15e not found: ID does not exist" containerID="f21405aa39c19bc9094da49c4dad6fd24354556cc9d4aa70b1d2126608e8c15e"
Sep 18 20:02:40.252289 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:02:40.252214    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f21405aa39c19bc9094da49c4dad6fd24354556cc9d4aa70b1d2126608e8c15e"} err="failed to get container status \"f21405aa39c19bc9094da49c4dad6fd24354556cc9d4aa70b1d2126608e8c15e\": rpc error: code = NotFound desc = could not find container \"f21405aa39c19bc9094da49c4dad6fd24354556cc9d4aa70b1d2126608e8c15e\": container with ID starting with f21405aa39c19bc9094da49c4dad6fd24354556cc9d4aa70b1d2126608e8c15e not found: ID does not exist"
Sep 18 20:02:40.263457 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:02:40.263424    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5t9j29"]
Sep 18 20:02:40.266908 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:02:40.266880    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5t9j29"]
Sep 18 20:02:40.318398 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:02:40.318357    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zf28t\" (UniqueName: \"kubernetes.io/projected/cb87e91c-c74f-4264-abe2-4831658c045b-kube-api-access-zf28t\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:02:40.318398 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:02:40.318390    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb87e91c-c74f-4264-abe2-4831658c045b-tls-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:02:42.000370 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:02:42.000334    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb87e91c-c74f-4264-abe2-4831658c045b" path="/var/lib/kubelet/pods/cb87e91c-c74f-4264-abe2-4831658c045b/volumes"
Sep 18 20:02:43.112876 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:02:43.112845    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-8lvwj"
Sep 18 20:03:04.003197 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:03:04.003167    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-58vl4_945a444e-9381-4e2f-b4c1-64be0ec88e6c/console-operator/2.log"
Sep 18 20:03:04.006304 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:03:04.006280    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-58vl4_945a444e-9381-4e2f-b4c1-64be0ec88e6c/console-operator/2.log"
Sep 18 20:03:19.316724 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:03:19.316681    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:03:19.417484 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:03:19.417440    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:03:23.917771 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:03:23.917734    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:03:24.170485 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:03:24.170393    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:03:25.324010 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:03:25.323946    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:03:30.845253 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:03:30.845155    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:03:30.900638 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:03:30.900608    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:03:32.438718 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:03:32.438668    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:03:52.584369 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:03:52.584324    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:03:58.149477 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:03:58.149436    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:03:58.249593 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:03:58.249552    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:04:16.122569 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:04:16.122536    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-8lvwj"]
Sep 18 20:04:16.130480 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:04:16.130447    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:04:16.207318 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:04:16.207282    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:04:23.113501 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:04:23.113466    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-8lvwj" podUID="a0578992-b41a-4605-b3a4-8d25c215f74b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:04:31.143858 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:04:31.143795    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-8lvwj" podUID="a0578992-b41a-4605-b3a4-8d25c215f74b" containerName="main" containerID="cri-o://09965a64e9cb49722d1e647208c853ef83a0acade4764bf563f503bd6c531d3d" gracePeriod=45
Sep 18 20:04:33.112637 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:04:33.112594    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-8lvwj" podUID="a0578992-b41a-4605-b3a4-8d25c215f74b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:04:43.113291 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:04:43.113249    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-8lvwj" podUID="a0578992-b41a-4605-b3a4-8d25c215f74b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:04:43.113793 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:04:43.113380    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-8lvwj"
Sep 18 20:04:48.210689 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:04:48.210645    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:04:48.304432 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:04:48.304393    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:04:51.375578 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:04:51.375539    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:04:51.580676 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:04:51.580640    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:04:53.112790 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:04:53.112756    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-8lvwj" podUID="a0578992-b41a-4605-b3a4-8d25c215f74b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:04:54.247311 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:04:54.247276    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:04:58.112015 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:04:58.111916    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:04:59.694249 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:04:59.694203    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:04:59.755860 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:04:59.755811    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:05:01.401721 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:01.401692    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-8lvwj"
Sep 18 20:05:01.505188 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:01.505081    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7ndqs\" (UniqueName: \"kubernetes.io/projected/a0578992-b41a-4605-b3a4-8d25c215f74b-kube-api-access-7ndqs\") pod \"a0578992-b41a-4605-b3a4-8d25c215f74b\" (UID: \"a0578992-b41a-4605-b3a4-8d25c215f74b\") "
Sep 18 20:05:01.505377 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:01.505195    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0578992-b41a-4605-b3a4-8d25c215f74b-tls-certs\") pod \"a0578992-b41a-4605-b3a4-8d25c215f74b\" (UID: \"a0578992-b41a-4605-b3a4-8d25c215f74b\") "
Sep 18 20:05:01.507435 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:01.507401    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0578992-b41a-4605-b3a4-8d25c215f74b-kube-api-access-7ndqs" (OuterVolumeSpecName: "kube-api-access-7ndqs") pod "a0578992-b41a-4605-b3a4-8d25c215f74b" (UID: "a0578992-b41a-4605-b3a4-8d25c215f74b"). InnerVolumeSpecName "kube-api-access-7ndqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:05:01.507546 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:01.507465    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0578992-b41a-4605-b3a4-8d25c215f74b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a0578992-b41a-4605-b3a4-8d25c215f74b" (UID: "a0578992-b41a-4605-b3a4-8d25c215f74b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:05:01.605860 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:01.605814    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0578992-b41a-4605-b3a4-8d25c215f74b-tls-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:05:01.605860 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:01.605852    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ndqs\" (UniqueName: \"kubernetes.io/projected/a0578992-b41a-4605-b3a4-8d25c215f74b-kube-api-access-7ndqs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:05:01.948510 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:01.948479    2564 generic.go:356] "Generic (PLEG): container finished" podID="a0578992-b41a-4605-b3a4-8d25c215f74b" containerID="09965a64e9cb49722d1e647208c853ef83a0acade4764bf563f503bd6c531d3d" exitCode=0
Sep 18 20:05:01.948680 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:01.948522    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-8lvwj"
Sep 18 20:05:01.948680 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:01.948563    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-8lvwj" event={"ID":"a0578992-b41a-4605-b3a4-8d25c215f74b","Type":"ContainerDied","Data":"09965a64e9cb49722d1e647208c853ef83a0acade4764bf563f503bd6c531d3d"}
Sep 18 20:05:01.948680 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:01.948617    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-8lvwj" event={"ID":"a0578992-b41a-4605-b3a4-8d25c215f74b","Type":"ContainerDied","Data":"a2e53f9abfef8f69931026c27e2acc624b23b509a6ab44a93381c4fb1a70cd2f"}
Sep 18 20:05:01.948680 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:01.948640    2564 scope.go:117] "RemoveContainer" containerID="09965a64e9cb49722d1e647208c853ef83a0acade4764bf563f503bd6c531d3d"
Sep 18 20:05:01.957464 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:01.957443    2564 scope.go:117] "RemoveContainer" containerID="09965a64e9cb49722d1e647208c853ef83a0acade4764bf563f503bd6c531d3d"
Sep 18 20:05:01.957731 ip-10-0-128-161 kubenswrapper[2564]: E0918 20:05:01.957714    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09965a64e9cb49722d1e647208c853ef83a0acade4764bf563f503bd6c531d3d\": container with ID starting with 09965a64e9cb49722d1e647208c853ef83a0acade4764bf563f503bd6c531d3d not found: ID does not exist" containerID="09965a64e9cb49722d1e647208c853ef83a0acade4764bf563f503bd6c531d3d"
Sep 18 20:05:01.957781 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:01.957742    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09965a64e9cb49722d1e647208c853ef83a0acade4764bf563f503bd6c531d3d"} err="failed to get container status \"09965a64e9cb49722d1e647208c853ef83a0acade4764bf563f503bd6c531d3d\": rpc error: code = NotFound desc = could not find container \"09965a64e9cb49722d1e647208c853ef83a0acade4764bf563f503bd6c531d3d\": container with ID starting with 09965a64e9cb49722d1e647208c853ef83a0acade4764bf563f503bd6c531d3d not found: ID does not exist"
Sep 18 20:05:01.985355 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:01.985304    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-8lvwj"]
Sep 18 20:05:01.991416 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:01.991381    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-8lvwj"]
Sep 18 20:05:02.002096 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:02.002065    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0578992-b41a-4605-b3a4-8d25c215f74b" path="/var/lib/kubelet/pods/a0578992-b41a-4605-b3a4-8d25c215f74b/volumes"
Sep 18 20:05:06.015594 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:06.015562    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:05:06.113446 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:06.113388    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:05:06.214560 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:06.214525    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:05:12.101711 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:12.101666    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:05:16.001370 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:16.001341    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4hwbt"]
Sep 18 20:05:16.001740 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:16.001677    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0578992-b41a-4605-b3a4-8d25c215f74b" containerName="main"
Sep 18 20:05:16.001740 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:16.001689    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0578992-b41a-4605-b3a4-8d25c215f74b" containerName="main"
Sep 18 20:05:16.001740 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:16.001696    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb87e91c-c74f-4264-abe2-4831658c045b" containerName="main"
Sep 18 20:05:16.001740 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:16.001702    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb87e91c-c74f-4264-abe2-4831658c045b" containerName="main"
Sep 18 20:05:16.001871 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:16.001761    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb87e91c-c74f-4264-abe2-4831658c045b" containerName="main"
Sep 18 20:05:16.001871 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:16.001774    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0578992-b41a-4605-b3a4-8d25c215f74b" containerName="main"
Sep 18 20:05:16.005826 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:16.005802    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4hwbt"
Sep 18 20:05:16.008666 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:16.008637    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"canary-v1-kserve-self-signed-certs\""
Sep 18 20:05:16.008826 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:16.008637    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"openshift-service-ca.crt\""
Sep 18 20:05:16.008826 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:16.008767    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"kube-root-ca.crt\""
Sep 18 20:05:16.008826 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:16.008787    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"default-dockercfg-wvfgf\""
Sep 18 20:05:16.009411 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:16.009385    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4hwbt"]
Sep 18 20:05:16.015519 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:16.015485    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b2011e8-a73f-43bf-a9ea-fdd27402ac08-home\") pod \"canary-v1-kserve-65d58ccc9-4hwbt\" (UID: \"2b2011e8-a73f-43bf-a9ea-fdd27402ac08\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4hwbt"
Sep 18 20:05:16.016048 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:16.016025    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b2011e8-a73f-43bf-a9ea-fdd27402ac08-tls-certs\") pod \"canary-v1-kserve-65d58ccc9-4hwbt\" (UID: \"2b2011e8-a73f-43bf-a9ea-fdd27402ac08\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4hwbt"
Sep 18 20:05:16.016206 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:16.016190    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b2011e8-a73f-43bf-a9ea-fdd27402ac08-tmp-dir\") pod \"canary-v1-kserve-65d58ccc9-4hwbt\" (UID: \"2b2011e8-a73f-43bf-a9ea-fdd27402ac08\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4hwbt"
Sep 18 20:05:16.016374 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:16.016347    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg269\" (UniqueName: \"kubernetes.io/projected/2b2011e8-a73f-43bf-a9ea-fdd27402ac08-kube-api-access-qg269\") pod \"canary-v1-kserve-65d58ccc9-4hwbt\" (UID: \"2b2011e8-a73f-43bf-a9ea-fdd27402ac08\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4hwbt"
Sep 18 20:05:16.016450 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:16.016410    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b2011e8-a73f-43bf-a9ea-fdd27402ac08-dshm\") pod \"canary-v1-kserve-65d58ccc9-4hwbt\" (UID: \"2b2011e8-a73f-43bf-a9ea-fdd27402ac08\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4hwbt"
Sep 18 20:05:16.016508 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:16.016499    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b2011e8-a73f-43bf-a9ea-fdd27402ac08-model-cache\") pod \"canary-v1-kserve-65d58ccc9-4hwbt\" (UID: \"2b2011e8-a73f-43bf-a9ea-fdd27402ac08\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4hwbt"
Sep 18 20:05:16.023396 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:16.023363    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:05:16.117430 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:16.117387    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b2011e8-a73f-43bf-a9ea-fdd27402ac08-home\") pod \"canary-v1-kserve-65d58ccc9-4hwbt\" (UID: \"2b2011e8-a73f-43bf-a9ea-fdd27402ac08\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4hwbt"
Sep 18 20:05:16.117430 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:16.117431    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b2011e8-a73f-43bf-a9ea-fdd27402ac08-tls-certs\") pod \"canary-v1-kserve-65d58ccc9-4hwbt\" (UID: \"2b2011e8-a73f-43bf-a9ea-fdd27402ac08\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4hwbt"
Sep 18 20:05:16.117694 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:16.117458    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b2011e8-a73f-43bf-a9ea-fdd27402ac08-tmp-dir\") pod \"canary-v1-kserve-65d58ccc9-4hwbt\" (UID: \"2b2011e8-a73f-43bf-a9ea-fdd27402ac08\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4hwbt"
Sep 18 20:05:16.117694 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:16.117494    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg269\" (UniqueName: \"kubernetes.io/projected/2b2011e8-a73f-43bf-a9ea-fdd27402ac08-kube-api-access-qg269\") pod \"canary-v1-kserve-65d58ccc9-4hwbt\" (UID: \"2b2011e8-a73f-43bf-a9ea-fdd27402ac08\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4hwbt"
Sep 18 20:05:16.117694 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:16.117526    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b2011e8-a73f-43bf-a9ea-fdd27402ac08-dshm\") pod \"canary-v1-kserve-65d58ccc9-4hwbt\" (UID: \"2b2011e8-a73f-43bf-a9ea-fdd27402ac08\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4hwbt"
Sep 18 20:05:16.117694 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:16.117600    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b2011e8-a73f-43bf-a9ea-fdd27402ac08-model-cache\") pod \"canary-v1-kserve-65d58ccc9-4hwbt\" (UID: \"2b2011e8-a73f-43bf-a9ea-fdd27402ac08\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4hwbt"
Sep 18 20:05:16.117919 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:16.117829    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b2011e8-a73f-43bf-a9ea-fdd27402ac08-home\") pod \"canary-v1-kserve-65d58ccc9-4hwbt\" (UID: \"2b2011e8-a73f-43bf-a9ea-fdd27402ac08\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4hwbt"
Sep 18 20:05:16.117919 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:16.117868    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b2011e8-a73f-43bf-a9ea-fdd27402ac08-tmp-dir\") pod \"canary-v1-kserve-65d58ccc9-4hwbt\" (UID: \"2b2011e8-a73f-43bf-a9ea-fdd27402ac08\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4hwbt"
Sep 18 20:05:16.118055 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:16.117924    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b2011e8-a73f-43bf-a9ea-fdd27402ac08-model-cache\") pod \"canary-v1-kserve-65d58ccc9-4hwbt\" (UID: \"2b2011e8-a73f-43bf-a9ea-fdd27402ac08\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4hwbt"
Sep 18 20:05:16.119922 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:16.119893    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b2011e8-a73f-43bf-a9ea-fdd27402ac08-dshm\") pod \"canary-v1-kserve-65d58ccc9-4hwbt\" (UID: \"2b2011e8-a73f-43bf-a9ea-fdd27402ac08\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4hwbt"
Sep 18 20:05:16.120149 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:16.120132    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b2011e8-a73f-43bf-a9ea-fdd27402ac08-tls-certs\") pod \"canary-v1-kserve-65d58ccc9-4hwbt\" (UID: \"2b2011e8-a73f-43bf-a9ea-fdd27402ac08\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4hwbt"
Sep 18 20:05:16.124974 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:16.124947    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg269\" (UniqueName: \"kubernetes.io/projected/2b2011e8-a73f-43bf-a9ea-fdd27402ac08-kube-api-access-qg269\") pod \"canary-v1-kserve-65d58ccc9-4hwbt\" (UID: \"2b2011e8-a73f-43bf-a9ea-fdd27402ac08\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4hwbt"
Sep 18 20:05:16.320223 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:16.320135    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4hwbt"
Sep 18 20:05:16.594897 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:16.594754    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4hwbt"]
Sep 18 20:05:16.597662 ip-10-0-128-161 kubenswrapper[2564]: W0918 20:05:16.597630    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b2011e8_a73f_43bf_a9ea_fdd27402ac08.slice/crio-261d89b64f2843de7fca0a167184624f415102164a9a397b43b7966a0b351c7d WatchSource:0}: Error finding container 261d89b64f2843de7fca0a167184624f415102164a9a397b43b7966a0b351c7d: Status 404 returned error can't find the container with id 261d89b64f2843de7fca0a167184624f415102164a9a397b43b7966a0b351c7d
Sep 18 20:05:16.599591 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:16.599574    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 20:05:17.022214 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:17.022183    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4hwbt" event={"ID":"2b2011e8-a73f-43bf-a9ea-fdd27402ac08","Type":"ContainerStarted","Data":"261d89b64f2843de7fca0a167184624f415102164a9a397b43b7966a0b351c7d"}
Sep 18 20:05:17.679917 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:17.679888    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:05:17.783375 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:17.783342    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:05:18.028470 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:18.028373    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4hwbt" event={"ID":"2b2011e8-a73f-43bf-a9ea-fdd27402ac08","Type":"ContainerStarted","Data":"0eae121fb99e862dd24e3f81676d543c47dfc97119739efabe614fbe56238b0b"}
Sep 18 20:05:18.047256 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:18.047187    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4hwbt" podStartSLOduration=1.973841832 podStartE2EDuration="3.047163342s" totalImagesPullingTime="1.07332151s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:05:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:05:18.044845742 +0000 UTC m=+1634.555201861" watchObservedRunningTime="2026-09-18 20:05:18.047163342 +0000 UTC m=+1634.557519436"
Sep 18 20:05:18.321420 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:18.321308    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4hwbt"
Sep 18 20:05:26.320967 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:26.320925    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4hwbt"
Sep 18 20:05:26.333944 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:26.333913    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4hwbt"
Sep 18 20:05:27.090253 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:27.090216    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4hwbt"
Sep 18 20:05:37.699503 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:37.699466    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-kr7v9"]
Sep 18 20:05:37.702249 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:37.702226    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-kr7v9"
Sep 18 20:05:37.704293 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:37.704270    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"canary-v2-kserve-self-signed-certs\""
Sep 18 20:05:37.713507 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:37.713481    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-kr7v9"]
Sep 18 20:05:37.732376 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:37.732342    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:05:37.823871 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:37.823829    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp8cl\" (UniqueName: \"kubernetes.io/projected/db4f7fec-dab0-44ba-952b-6c757f76024f-kube-api-access-pp8cl\") pod \"canary-v2-kserve-7f65dbbccc-kr7v9\" (UID: \"db4f7fec-dab0-44ba-952b-6c757f76024f\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-kr7v9"
Sep 18 20:05:37.823871 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:37.823867    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db4f7fec-dab0-44ba-952b-6c757f76024f-home\") pod \"canary-v2-kserve-7f65dbbccc-kr7v9\" (UID: \"db4f7fec-dab0-44ba-952b-6c757f76024f\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-kr7v9"
Sep 18 20:05:37.824139 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:37.823891    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db4f7fec-dab0-44ba-952b-6c757f76024f-tls-certs\") pod \"canary-v2-kserve-7f65dbbccc-kr7v9\" (UID: \"db4f7fec-dab0-44ba-952b-6c757f76024f\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-kr7v9"
Sep 18 20:05:37.824139 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:37.824015    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db4f7fec-dab0-44ba-952b-6c757f76024f-model-cache\") pod \"canary-v2-kserve-7f65dbbccc-kr7v9\" (UID: \"db4f7fec-dab0-44ba-952b-6c757f76024f\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-kr7v9"
Sep 18 20:05:37.824139 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:37.824096    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db4f7fec-dab0-44ba-952b-6c757f76024f-dshm\") pod \"canary-v2-kserve-7f65dbbccc-kr7v9\" (UID: \"db4f7fec-dab0-44ba-952b-6c757f76024f\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-kr7v9"
Sep 18 20:05:37.824139 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:37.824130    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db4f7fec-dab0-44ba-952b-6c757f76024f-tmp-dir\") pod \"canary-v2-kserve-7f65dbbccc-kr7v9\" (UID: \"db4f7fec-dab0-44ba-952b-6c757f76024f\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-kr7v9"
Sep 18 20:05:37.925096 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:37.925053    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db4f7fec-dab0-44ba-952b-6c757f76024f-dshm\") pod \"canary-v2-kserve-7f65dbbccc-kr7v9\" (UID: \"db4f7fec-dab0-44ba-952b-6c757f76024f\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-kr7v9"
Sep 18 20:05:37.925096 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:37.925103    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db4f7fec-dab0-44ba-952b-6c757f76024f-tmp-dir\") pod \"canary-v2-kserve-7f65dbbccc-kr7v9\" (UID: \"db4f7fec-dab0-44ba-952b-6c757f76024f\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-kr7v9"
Sep 18 20:05:37.925351 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:37.925143    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp8cl\" (UniqueName: \"kubernetes.io/projected/db4f7fec-dab0-44ba-952b-6c757f76024f-kube-api-access-pp8cl\") pod \"canary-v2-kserve-7f65dbbccc-kr7v9\" (UID: \"db4f7fec-dab0-44ba-952b-6c757f76024f\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-kr7v9"
Sep 18 20:05:37.925351 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:37.925163    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db4f7fec-dab0-44ba-952b-6c757f76024f-home\") pod \"canary-v2-kserve-7f65dbbccc-kr7v9\" (UID: \"db4f7fec-dab0-44ba-952b-6c757f76024f\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-kr7v9"
Sep 18 20:05:37.925351 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:37.925183    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db4f7fec-dab0-44ba-952b-6c757f76024f-tls-certs\") pod \"canary-v2-kserve-7f65dbbccc-kr7v9\" (UID: \"db4f7fec-dab0-44ba-952b-6c757f76024f\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-kr7v9"
Sep 18 20:05:37.925351 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:37.925210    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db4f7fec-dab0-44ba-952b-6c757f76024f-model-cache\") pod \"canary-v2-kserve-7f65dbbccc-kr7v9\" (UID: \"db4f7fec-dab0-44ba-952b-6c757f76024f\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-kr7v9"
Sep 18 20:05:37.925667 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:37.925638    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db4f7fec-dab0-44ba-952b-6c757f76024f-tmp-dir\") pod \"canary-v2-kserve-7f65dbbccc-kr7v9\" (UID: \"db4f7fec-dab0-44ba-952b-6c757f76024f\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-kr7v9"
Sep 18 20:05:37.925667 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:37.925654    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db4f7fec-dab0-44ba-952b-6c757f76024f-home\") pod \"canary-v2-kserve-7f65dbbccc-kr7v9\" (UID: \"db4f7fec-dab0-44ba-952b-6c757f76024f\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-kr7v9"
Sep 18 20:05:37.925809 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:37.925691    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db4f7fec-dab0-44ba-952b-6c757f76024f-model-cache\") pod \"canary-v2-kserve-7f65dbbccc-kr7v9\" (UID: \"db4f7fec-dab0-44ba-952b-6c757f76024f\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-kr7v9"
Sep 18 20:05:37.927512 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:37.927492    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db4f7fec-dab0-44ba-952b-6c757f76024f-dshm\") pod \"canary-v2-kserve-7f65dbbccc-kr7v9\" (UID: \"db4f7fec-dab0-44ba-952b-6c757f76024f\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-kr7v9"
Sep 18 20:05:37.927839 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:37.927821    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db4f7fec-dab0-44ba-952b-6c757f76024f-tls-certs\") pod \"canary-v2-kserve-7f65dbbccc-kr7v9\" (UID: \"db4f7fec-dab0-44ba-952b-6c757f76024f\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-kr7v9"
Sep 18 20:05:37.932278 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:37.932251    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp8cl\" (UniqueName: \"kubernetes.io/projected/db4f7fec-dab0-44ba-952b-6c757f76024f-kube-api-access-pp8cl\") pod \"canary-v2-kserve-7f65dbbccc-kr7v9\" (UID: \"db4f7fec-dab0-44ba-952b-6c757f76024f\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-kr7v9"
Sep 18 20:05:38.013381 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:38.013283    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-kr7v9"
Sep 18 20:05:38.286096 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:38.286069    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-kr7v9"]
Sep 18 20:05:38.289334 ip-10-0-128-161 kubenswrapper[2564]: W0918 20:05:38.289298    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb4f7fec_dab0_44ba_952b_6c757f76024f.slice/crio-ffec2b78388c87f5c1c597983dd12cba0c8dacd342e4528628b3f27b16724edf WatchSource:0}: Error finding container ffec2b78388c87f5c1c597983dd12cba0c8dacd342e4528628b3f27b16724edf: Status 404 returned error can't find the container with id ffec2b78388c87f5c1c597983dd12cba0c8dacd342e4528628b3f27b16724edf
Sep 18 20:05:39.146348 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:39.146309    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-kr7v9" event={"ID":"db4f7fec-dab0-44ba-952b-6c757f76024f","Type":"ContainerStarted","Data":"a15b5d1eaf67b155127f99e53988bcbf0104046672f88ef5df4b7c1573292b15"}
Sep 18 20:05:39.146748 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:39.146370    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-kr7v9" event={"ID":"db4f7fec-dab0-44ba-952b-6c757f76024f","Type":"ContainerStarted","Data":"ffec2b78388c87f5c1c597983dd12cba0c8dacd342e4528628b3f27b16724edf"}
Sep 18 20:05:39.162812 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:39.162761    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-kr7v9" podStartSLOduration=2.162744633 podStartE2EDuration="2.162744633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:05:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:05:39.161200415 +0000 UTC m=+1655.671556519" watchObservedRunningTime="2026-09-18 20:05:39.162744633 +0000 UTC m=+1655.673100764"
Sep 18 20:05:40.014138 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:40.014109    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-kr7v9"
Sep 18 20:05:48.014041 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:48.014008    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-kr7v9"
Sep 18 20:05:48.027019 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:48.026976    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-kr7v9"
Sep 18 20:05:48.204053 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:05:48.204020    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-kr7v9"
Sep 18 20:06:04.005459 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:06:04.005419    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:04.061558 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:06:04.061520    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:16.722171 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:06:16.722127    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:51.502732 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:06:51.502688    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:51.595584 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:06:51.595544    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:53.046241 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:06:53.046202    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:53.147106 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:06:53.147067    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:55.061574 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:06:55.061536    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:55.164184 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:06:55.164154    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:57.415299 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:06:57.415260    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:07:02.837355 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:02.837313    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:07:02.899784 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:02.899752    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:07:05.494928 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:05.494890    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:07:05.671870 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:05.671834    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedf46wf"]
Sep 18 20:07:05.675312 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:05.675293    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedf46wf"
Sep 18 20:07:05.678712 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:05.678686    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Sep 18 20:07:05.678869 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:05.678718    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-brqps\""
Sep 18 20:07:05.678869 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:05.678689    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Sep 18 20:07:05.678869 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:05.678693    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Sep 18 20:07:05.678869 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:05.678807    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-epp-sa-dockercfg-n8snm\""
Sep 18 20:07:05.684124 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:05.684101    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedf46wf"]
Sep 18 20:07:05.696924 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:05.696888    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:07:05.740635 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:05.740590    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e71b25c5-f3cb-4ed4-b4c3-aa0efae85b6c-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedf46wf\" (UID: \"e71b25c5-f3cb-4ed4-b4c3-aa0efae85b6c\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedf46wf"
Sep 18 20:07:05.740821 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:05.740669    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldz2d\" (UniqueName: \"kubernetes.io/projected/e71b25c5-f3cb-4ed4-b4c3-aa0efae85b6c-kube-api-access-ldz2d\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedf46wf\" (UID: \"e71b25c5-f3cb-4ed4-b4c3-aa0efae85b6c\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedf46wf"
Sep 18 20:07:05.841601 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:05.841507    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e71b25c5-f3cb-4ed4-b4c3-aa0efae85b6c-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedf46wf\" (UID: \"e71b25c5-f3cb-4ed4-b4c3-aa0efae85b6c\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedf46wf"
Sep 18 20:07:05.841601 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:05.841594    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldz2d\" (UniqueName: \"kubernetes.io/projected/e71b25c5-f3cb-4ed4-b4c3-aa0efae85b6c-kube-api-access-ldz2d\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedf46wf\" (UID: \"e71b25c5-f3cb-4ed4-b4c3-aa0efae85b6c\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedf46wf"
Sep 18 20:07:05.844199 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:05.844164    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e71b25c5-f3cb-4ed4-b4c3-aa0efae85b6c-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedf46wf\" (UID: \"e71b25c5-f3cb-4ed4-b4c3-aa0efae85b6c\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedf46wf"
Sep 18 20:07:05.850037 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:05.850004    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldz2d\" (UniqueName: \"kubernetes.io/projected/e71b25c5-f3cb-4ed4-b4c3-aa0efae85b6c-kube-api-access-ldz2d\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedf46wf\" (UID: \"e71b25c5-f3cb-4ed4-b4c3-aa0efae85b6c\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedf46wf"
Sep 18 20:07:05.986743 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:05.986698    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedf46wf"
Sep 18 20:07:06.262174 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:06.262136    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedf46wf"]
Sep 18 20:07:06.266190 ip-10-0-128-161 kubenswrapper[2564]: W0918 20:07:06.266152    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode71b25c5_f3cb_4ed4_b4c3_aa0efae85b6c.slice/crio-2c986d6b4bd55660d4e6d9600a683e98ac95e7a8f09495238bfaac0b1a75c5cb WatchSource:0}: Error finding container 2c986d6b4bd55660d4e6d9600a683e98ac95e7a8f09495238bfaac0b1a75c5cb: Status 404 returned error can't find the container with id 2c986d6b4bd55660d4e6d9600a683e98ac95e7a8f09495238bfaac0b1a75c5cb
Sep 18 20:07:06.579560 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:06.579469    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedf46wf" event={"ID":"e71b25c5-f3cb-4ed4-b4c3-aa0efae85b6c","Type":"ContainerStarted","Data":"d0fac324034c853f1cc7e0ae7e136908d4e09df7f1da06437168aac4b6d4c4b8"}
Sep 18 20:07:06.579560 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:06.579532    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedf46wf" event={"ID":"e71b25c5-f3cb-4ed4-b4c3-aa0efae85b6c","Type":"ContainerStarted","Data":"2c986d6b4bd55660d4e6d9600a683e98ac95e7a8f09495238bfaac0b1a75c5cb"}
Sep 18 20:07:06.579961 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:06.579648    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedf46wf"
Sep 18 20:07:06.606848 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:06.606789    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedf46wf" podStartSLOduration=1.60677299 podStartE2EDuration="1.60677299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:07:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:07:06.603890969 +0000 UTC m=+1743.114247083" watchObservedRunningTime="2026-09-18 20:07:06.60677299 +0000 UTC m=+1743.117129080"
Sep 18 20:07:22.824493 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:22.824378    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:07:36.149231 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:36.149196    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:07:36.243954 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:36.243915    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:07:36.358655 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:36.358610    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:07:36.400917 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:36.400831    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4hwbt"]
Sep 18 20:07:36.516489 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:36.516457    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:07:37.010208 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:37.010168    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-kr7v9"]
Sep 18 20:07:37.024303 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:37.024233    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:07:37.587078 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:37.587046    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedf46wf"
Sep 18 20:07:41.231518 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:41.231484    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:07:41.440176 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:41.440134    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:07:49.818741 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:49.818706    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:07:51.420467 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:51.420398    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4hwbt" podUID="2b2011e8-a73f-43bf-a9ea-fdd27402ac08" containerName="main" containerID="cri-o://0eae121fb99e862dd24e3f81676d543c47dfc97119739efabe614fbe56238b0b" gracePeriod=45
Sep 18 20:07:51.670620 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:51.670556    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4hwbt"
Sep 18 20:07:51.728048 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:51.728011    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b2011e8-a73f-43bf-a9ea-fdd27402ac08-model-cache\") pod \"2b2011e8-a73f-43bf-a9ea-fdd27402ac08\" (UID: \"2b2011e8-a73f-43bf-a9ea-fdd27402ac08\") "
Sep 18 20:07:51.728048 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:51.728049    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qg269\" (UniqueName: \"kubernetes.io/projected/2b2011e8-a73f-43bf-a9ea-fdd27402ac08-kube-api-access-qg269\") pod \"2b2011e8-a73f-43bf-a9ea-fdd27402ac08\" (UID: \"2b2011e8-a73f-43bf-a9ea-fdd27402ac08\") "
Sep 18 20:07:51.728310 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:51.728075    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b2011e8-a73f-43bf-a9ea-fdd27402ac08-tls-certs\") pod \"2b2011e8-a73f-43bf-a9ea-fdd27402ac08\" (UID: \"2b2011e8-a73f-43bf-a9ea-fdd27402ac08\") "
Sep 18 20:07:51.728310 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:51.728098    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b2011e8-a73f-43bf-a9ea-fdd27402ac08-tmp-dir\") pod \"2b2011e8-a73f-43bf-a9ea-fdd27402ac08\" (UID: \"2b2011e8-a73f-43bf-a9ea-fdd27402ac08\") "
Sep 18 20:07:51.728310 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:51.728147    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b2011e8-a73f-43bf-a9ea-fdd27402ac08-dshm\") pod \"2b2011e8-a73f-43bf-a9ea-fdd27402ac08\" (UID: \"2b2011e8-a73f-43bf-a9ea-fdd27402ac08\") "
Sep 18 20:07:51.728310 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:51.728174    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b2011e8-a73f-43bf-a9ea-fdd27402ac08-home\") pod \"2b2011e8-a73f-43bf-a9ea-fdd27402ac08\" (UID: \"2b2011e8-a73f-43bf-a9ea-fdd27402ac08\") "
Sep 18 20:07:51.728521 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:51.728328    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b2011e8-a73f-43bf-a9ea-fdd27402ac08-model-cache" (OuterVolumeSpecName: "model-cache") pod "2b2011e8-a73f-43bf-a9ea-fdd27402ac08" (UID: "2b2011e8-a73f-43bf-a9ea-fdd27402ac08"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:07:51.728521 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:51.728408    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b2011e8-a73f-43bf-a9ea-fdd27402ac08-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2b2011e8-a73f-43bf-a9ea-fdd27402ac08" (UID: "2b2011e8-a73f-43bf-a9ea-fdd27402ac08"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:07:51.728595 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:51.728522    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b2011e8-a73f-43bf-a9ea-fdd27402ac08-home" (OuterVolumeSpecName: "home") pod "2b2011e8-a73f-43bf-a9ea-fdd27402ac08" (UID: "2b2011e8-a73f-43bf-a9ea-fdd27402ac08"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:07:51.728595 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:51.728537    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b2011e8-a73f-43bf-a9ea-fdd27402ac08-model-cache\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:07:51.728595 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:51.728562    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b2011e8-a73f-43bf-a9ea-fdd27402ac08-tmp-dir\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:07:51.730536 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:51.730507    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b2011e8-a73f-43bf-a9ea-fdd27402ac08-dshm" (OuterVolumeSpecName: "dshm") pod "2b2011e8-a73f-43bf-a9ea-fdd27402ac08" (UID: "2b2011e8-a73f-43bf-a9ea-fdd27402ac08"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:07:51.730536 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:51.730528    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b2011e8-a73f-43bf-a9ea-fdd27402ac08-kube-api-access-qg269" (OuterVolumeSpecName: "kube-api-access-qg269") pod "2b2011e8-a73f-43bf-a9ea-fdd27402ac08" (UID: "2b2011e8-a73f-43bf-a9ea-fdd27402ac08"). InnerVolumeSpecName "kube-api-access-qg269". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:07:51.730676 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:51.730517    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b2011e8-a73f-43bf-a9ea-fdd27402ac08-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2b2011e8-a73f-43bf-a9ea-fdd27402ac08" (UID: "2b2011e8-a73f-43bf-a9ea-fdd27402ac08"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:07:51.813226 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:51.813191    2564 generic.go:356] "Generic (PLEG): container finished" podID="2b2011e8-a73f-43bf-a9ea-fdd27402ac08" containerID="0eae121fb99e862dd24e3f81676d543c47dfc97119739efabe614fbe56238b0b" exitCode=0
Sep 18 20:07:51.813426 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:51.813237    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4hwbt"
Sep 18 20:07:51.813426 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:51.813245    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4hwbt" event={"ID":"2b2011e8-a73f-43bf-a9ea-fdd27402ac08","Type":"ContainerDied","Data":"0eae121fb99e862dd24e3f81676d543c47dfc97119739efabe614fbe56238b0b"}
Sep 18 20:07:51.813426 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:51.813287    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4hwbt" event={"ID":"2b2011e8-a73f-43bf-a9ea-fdd27402ac08","Type":"ContainerDied","Data":"261d89b64f2843de7fca0a167184624f415102164a9a397b43b7966a0b351c7d"}
Sep 18 20:07:51.813426 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:51.813308    2564 scope.go:117] "RemoveContainer" containerID="0eae121fb99e862dd24e3f81676d543c47dfc97119739efabe614fbe56238b0b"
Sep 18 20:07:51.822692 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:51.822673    2564 scope.go:117] "RemoveContainer" containerID="0eae121fb99e862dd24e3f81676d543c47dfc97119739efabe614fbe56238b0b"
Sep 18 20:07:51.823066 ip-10-0-128-161 kubenswrapper[2564]: E0918 20:07:51.823045    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0eae121fb99e862dd24e3f81676d543c47dfc97119739efabe614fbe56238b0b\": container with ID starting with 0eae121fb99e862dd24e3f81676d543c47dfc97119739efabe614fbe56238b0b not found: ID does not exist" containerID="0eae121fb99e862dd24e3f81676d543c47dfc97119739efabe614fbe56238b0b"
Sep 18 20:07:51.823129 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:51.823078    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0eae121fb99e862dd24e3f81676d543c47dfc97119739efabe614fbe56238b0b"} err="failed to get container status \"0eae121fb99e862dd24e3f81676d543c47dfc97119739efabe614fbe56238b0b\": rpc error: code = NotFound desc = could not find container \"0eae121fb99e862dd24e3f81676d543c47dfc97119739efabe614fbe56238b0b\": container with ID starting with 0eae121fb99e862dd24e3f81676d543c47dfc97119739efabe614fbe56238b0b not found: ID does not exist"
Sep 18 20:07:51.829275 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:51.829250    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qg269\" (UniqueName: \"kubernetes.io/projected/2b2011e8-a73f-43bf-a9ea-fdd27402ac08-kube-api-access-qg269\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:07:51.829275 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:51.829277    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b2011e8-a73f-43bf-a9ea-fdd27402ac08-tls-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:07:51.829469 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:51.829286    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b2011e8-a73f-43bf-a9ea-fdd27402ac08-dshm\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:07:51.829469 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:51.829294    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b2011e8-a73f-43bf-a9ea-fdd27402ac08-home\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:07:51.834451 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:51.834424    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4hwbt"]
Sep 18 20:07:51.837792 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:51.837764    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-4hwbt"]
Sep 18 20:07:51.920445 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:51.920412    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedf46wf"]
Sep 18 20:07:51.923018 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:51.922904    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:07:52.000993 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:52.000943    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b2011e8-a73f-43bf-a9ea-fdd27402ac08" path="/var/lib/kubelet/pods/2b2011e8-a73f-43bf-a9ea-fdd27402ac08/volumes"
Sep 18 20:07:52.022018 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:52.021971    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:07:52.030733 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:52.030698    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-kr7v9" podUID="db4f7fec-dab0-44ba-952b-6c757f76024f" containerName="main" containerID="cri-o://a15b5d1eaf67b155127f99e53988bcbf0104046672f88ef5df4b7c1573292b15" gracePeriod=45
Sep 18 20:07:52.193665 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:52.193566    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-kr7v9" podUID="db4f7fec-dab0-44ba-952b-6c757f76024f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 18 20:07:52.285659 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:52.285633    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-kr7v9"
Sep 18 20:07:52.334076 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:52.334044    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pp8cl\" (UniqueName: \"kubernetes.io/projected/db4f7fec-dab0-44ba-952b-6c757f76024f-kube-api-access-pp8cl\") pod \"db4f7fec-dab0-44ba-952b-6c757f76024f\" (UID: \"db4f7fec-dab0-44ba-952b-6c757f76024f\") "
Sep 18 20:07:52.334261 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:52.334093    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db4f7fec-dab0-44ba-952b-6c757f76024f-model-cache\") pod \"db4f7fec-dab0-44ba-952b-6c757f76024f\" (UID: \"db4f7fec-dab0-44ba-952b-6c757f76024f\") "
Sep 18 20:07:52.334261 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:52.334115    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db4f7fec-dab0-44ba-952b-6c757f76024f-dshm\") pod \"db4f7fec-dab0-44ba-952b-6c757f76024f\" (UID: \"db4f7fec-dab0-44ba-952b-6c757f76024f\") "
Sep 18 20:07:52.334261 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:52.334147    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db4f7fec-dab0-44ba-952b-6c757f76024f-tmp-dir\") pod \"db4f7fec-dab0-44ba-952b-6c757f76024f\" (UID: \"db4f7fec-dab0-44ba-952b-6c757f76024f\") "
Sep 18 20:07:52.334261 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:52.334235    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db4f7fec-dab0-44ba-952b-6c757f76024f-tls-certs\") pod \"db4f7fec-dab0-44ba-952b-6c757f76024f\" (UID: \"db4f7fec-dab0-44ba-952b-6c757f76024f\") "
Sep 18 20:07:52.334485 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:52.334338    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db4f7fec-dab0-44ba-952b-6c757f76024f-home\") pod \"db4f7fec-dab0-44ba-952b-6c757f76024f\" (UID: \"db4f7fec-dab0-44ba-952b-6c757f76024f\") "
Sep 18 20:07:52.334485 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:52.334352    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db4f7fec-dab0-44ba-952b-6c757f76024f-model-cache" (OuterVolumeSpecName: "model-cache") pod "db4f7fec-dab0-44ba-952b-6c757f76024f" (UID: "db4f7fec-dab0-44ba-952b-6c757f76024f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:07:52.334485 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:52.334375    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db4f7fec-dab0-44ba-952b-6c757f76024f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "db4f7fec-dab0-44ba-952b-6c757f76024f" (UID: "db4f7fec-dab0-44ba-952b-6c757f76024f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:07:52.334628 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:52.334616    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db4f7fec-dab0-44ba-952b-6c757f76024f-model-cache\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:07:52.334687 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:52.334622    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db4f7fec-dab0-44ba-952b-6c757f76024f-home" (OuterVolumeSpecName: "home") pod "db4f7fec-dab0-44ba-952b-6c757f76024f" (UID: "db4f7fec-dab0-44ba-952b-6c757f76024f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:07:52.334687 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:52.334635    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db4f7fec-dab0-44ba-952b-6c757f76024f-tmp-dir\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:07:52.336663 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:52.336637    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db4f7fec-dab0-44ba-952b-6c757f76024f-dshm" (OuterVolumeSpecName: "dshm") pod "db4f7fec-dab0-44ba-952b-6c757f76024f" (UID: "db4f7fec-dab0-44ba-952b-6c757f76024f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:07:52.336812 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:52.336798    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db4f7fec-dab0-44ba-952b-6c757f76024f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "db4f7fec-dab0-44ba-952b-6c757f76024f" (UID: "db4f7fec-dab0-44ba-952b-6c757f76024f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:07:52.336858 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:52.336818    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db4f7fec-dab0-44ba-952b-6c757f76024f-kube-api-access-pp8cl" (OuterVolumeSpecName: "kube-api-access-pp8cl") pod "db4f7fec-dab0-44ba-952b-6c757f76024f" (UID: "db4f7fec-dab0-44ba-952b-6c757f76024f"). InnerVolumeSpecName "kube-api-access-pp8cl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:07:52.435741 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:52.435699    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db4f7fec-dab0-44ba-952b-6c757f76024f-tls-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:07:52.435741 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:52.435735    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db4f7fec-dab0-44ba-952b-6c757f76024f-home\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:07:52.435741 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:52.435750    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp8cl\" (UniqueName: \"kubernetes.io/projected/db4f7fec-dab0-44ba-952b-6c757f76024f-kube-api-access-pp8cl\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:07:52.436206 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:52.435761    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db4f7fec-dab0-44ba-952b-6c757f76024f-dshm\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:07:52.820658 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:52.820626    2564 generic.go:356] "Generic (PLEG): container finished" podID="db4f7fec-dab0-44ba-952b-6c757f76024f" containerID="a15b5d1eaf67b155127f99e53988bcbf0104046672f88ef5df4b7c1573292b15" exitCode=0
Sep 18 20:07:52.820929 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:52.820675    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-kr7v9" event={"ID":"db4f7fec-dab0-44ba-952b-6c757f76024f","Type":"ContainerDied","Data":"a15b5d1eaf67b155127f99e53988bcbf0104046672f88ef5df4b7c1573292b15"}
Sep 18 20:07:52.820929 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:52.820679    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-kr7v9"
Sep 18 20:07:52.820929 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:52.820717    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-kr7v9" event={"ID":"db4f7fec-dab0-44ba-952b-6c757f76024f","Type":"ContainerDied","Data":"ffec2b78388c87f5c1c597983dd12cba0c8dacd342e4528628b3f27b16724edf"}
Sep 18 20:07:52.820929 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:52.820742    2564 scope.go:117] "RemoveContainer" containerID="a15b5d1eaf67b155127f99e53988bcbf0104046672f88ef5df4b7c1573292b15"
Sep 18 20:07:52.833819 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:52.833797    2564 scope.go:117] "RemoveContainer" containerID="a15b5d1eaf67b155127f99e53988bcbf0104046672f88ef5df4b7c1573292b15"
Sep 18 20:07:52.834167 ip-10-0-128-161 kubenswrapper[2564]: E0918 20:07:52.834146    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a15b5d1eaf67b155127f99e53988bcbf0104046672f88ef5df4b7c1573292b15\": container with ID starting with a15b5d1eaf67b155127f99e53988bcbf0104046672f88ef5df4b7c1573292b15 not found: ID does not exist" containerID="a15b5d1eaf67b155127f99e53988bcbf0104046672f88ef5df4b7c1573292b15"
Sep 18 20:07:52.834223 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:52.834176    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a15b5d1eaf67b155127f99e53988bcbf0104046672f88ef5df4b7c1573292b15"} err="failed to get container status \"a15b5d1eaf67b155127f99e53988bcbf0104046672f88ef5df4b7c1573292b15\": rpc error: code = NotFound desc = could not find container \"a15b5d1eaf67b155127f99e53988bcbf0104046672f88ef5df4b7c1573292b15\": container with ID starting with a15b5d1eaf67b155127f99e53988bcbf0104046672f88ef5df4b7c1573292b15 not found: ID does not exist"
Sep 18 20:07:52.841526 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:52.841500    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-kr7v9"]
Sep 18 20:07:52.846374 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:52.846346    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-kr7v9"]
Sep 18 20:07:54.000027 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:53.999975    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db4f7fec-dab0-44ba-952b-6c757f76024f" path="/var/lib/kubelet/pods/db4f7fec-dab0-44ba-952b-6c757f76024f/volumes"
Sep 18 20:07:57.586879 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:07:57.586789    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedf46wf" podUID="e71b25c5-f3cb-4ed4-b4c3-aa0efae85b6c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:08:04.031809 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:04.031779    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-58vl4_945a444e-9381-4e2f-b4c1-64be0ec88e6c/console-operator/2.log"
Sep 18 20:08:04.035550 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:04.035527    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-58vl4_945a444e-9381-4e2f-b4c1-64be0ec88e6c/console-operator/2.log"
Sep 18 20:08:06.942116 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:06.942029    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedf46wf" podUID="e71b25c5-f3cb-4ed4-b4c3-aa0efae85b6c" containerName="main" containerID="cri-o://d0fac324034c853f1cc7e0ae7e136908d4e09df7f1da06437168aac4b6d4c4b8" gracePeriod=45
Sep 18 20:08:07.587023 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:07.586969    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedf46wf" podUID="e71b25c5-f3cb-4ed4-b4c3-aa0efae85b6c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:08:17.587069 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:17.587028    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedf46wf" podUID="e71b25c5-f3cb-4ed4-b4c3-aa0efae85b6c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:08:17.587468 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:17.587139    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedf46wf"
Sep 18 20:08:18.274955 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:18.274914    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:08:23.777699 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:23.777662    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:08:23.878555 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:23.878520    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:08:27.586739 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:27.586697    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedf46wf" podUID="e71b25c5-f3cb-4ed4-b4c3-aa0efae85b6c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:08:37.205347 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:37.205272    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedf46wf"
Sep 18 20:08:37.342009 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:37.341940    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ldz2d\" (UniqueName: \"kubernetes.io/projected/e71b25c5-f3cb-4ed4-b4c3-aa0efae85b6c-kube-api-access-ldz2d\") pod \"e71b25c5-f3cb-4ed4-b4c3-aa0efae85b6c\" (UID: \"e71b25c5-f3cb-4ed4-b4c3-aa0efae85b6c\") "
Sep 18 20:08:37.342202 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:37.342019    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e71b25c5-f3cb-4ed4-b4c3-aa0efae85b6c-tls-certs\") pod \"e71b25c5-f3cb-4ed4-b4c3-aa0efae85b6c\" (UID: \"e71b25c5-f3cb-4ed4-b4c3-aa0efae85b6c\") "
Sep 18 20:08:37.344424 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:37.344391    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e71b25c5-f3cb-4ed4-b4c3-aa0efae85b6c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e71b25c5-f3cb-4ed4-b4c3-aa0efae85b6c" (UID: "e71b25c5-f3cb-4ed4-b4c3-aa0efae85b6c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:08:37.344424 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:37.344405    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e71b25c5-f3cb-4ed4-b4c3-aa0efae85b6c-kube-api-access-ldz2d" (OuterVolumeSpecName: "kube-api-access-ldz2d") pod "e71b25c5-f3cb-4ed4-b4c3-aa0efae85b6c" (UID: "e71b25c5-f3cb-4ed4-b4c3-aa0efae85b6c"). InnerVolumeSpecName "kube-api-access-ldz2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:08:37.442779 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:37.442742    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldz2d\" (UniqueName: \"kubernetes.io/projected/e71b25c5-f3cb-4ed4-b4c3-aa0efae85b6c-kube-api-access-ldz2d\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:08:37.442779 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:37.442777    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e71b25c5-f3cb-4ed4-b4c3-aa0efae85b6c-tls-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:08:38.061464 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:38.061427    2564 generic.go:356] "Generic (PLEG): container finished" podID="e71b25c5-f3cb-4ed4-b4c3-aa0efae85b6c" containerID="d0fac324034c853f1cc7e0ae7e136908d4e09df7f1da06437168aac4b6d4c4b8" exitCode=0
Sep 18 20:08:38.061628 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:38.061473    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedf46wf"
Sep 18 20:08:38.061628 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:38.061509    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedf46wf" event={"ID":"e71b25c5-f3cb-4ed4-b4c3-aa0efae85b6c","Type":"ContainerDied","Data":"d0fac324034c853f1cc7e0ae7e136908d4e09df7f1da06437168aac4b6d4c4b8"}
Sep 18 20:08:38.061628 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:38.061563    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedf46wf" event={"ID":"e71b25c5-f3cb-4ed4-b4c3-aa0efae85b6c","Type":"ContainerDied","Data":"2c986d6b4bd55660d4e6d9600a683e98ac95e7a8f09495238bfaac0b1a75c5cb"}
Sep 18 20:08:38.061628 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:38.061584    2564 scope.go:117] "RemoveContainer" containerID="d0fac324034c853f1cc7e0ae7e136908d4e09df7f1da06437168aac4b6d4c4b8"
Sep 18 20:08:38.070731 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:38.070713    2564 scope.go:117] "RemoveContainer" containerID="d0fac324034c853f1cc7e0ae7e136908d4e09df7f1da06437168aac4b6d4c4b8"
Sep 18 20:08:38.071027 ip-10-0-128-161 kubenswrapper[2564]: E0918 20:08:38.071006    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0fac324034c853f1cc7e0ae7e136908d4e09df7f1da06437168aac4b6d4c4b8\": container with ID starting with d0fac324034c853f1cc7e0ae7e136908d4e09df7f1da06437168aac4b6d4c4b8 not found: ID does not exist" containerID="d0fac324034c853f1cc7e0ae7e136908d4e09df7f1da06437168aac4b6d4c4b8"
Sep 18 20:08:38.071084 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:38.071038    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0fac324034c853f1cc7e0ae7e136908d4e09df7f1da06437168aac4b6d4c4b8"} err="failed to get container status \"d0fac324034c853f1cc7e0ae7e136908d4e09df7f1da06437168aac4b6d4c4b8\": rpc error: code = NotFound desc = could not find container \"d0fac324034c853f1cc7e0ae7e136908d4e09df7f1da06437168aac4b6d4c4b8\": container with ID starting with d0fac324034c853f1cc7e0ae7e136908d4e09df7f1da06437168aac4b6d4c4b8 not found: ID does not exist"
Sep 18 20:08:38.076690 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:38.076664    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedf46wf"]
Sep 18 20:08:38.080657 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:38.080622    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedf46wf"]
Sep 18 20:08:38.738282 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:38.738250    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:08:38.838019 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:38.837958    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:08:38.938520 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:38.938484    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:08:39.020270 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:39.020184    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:08:39.434790 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:39.434741    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:08:40.000257 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:40.000213    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e71b25c5-f3cb-4ed4-b4c3-aa0efae85b6c" path="/var/lib/kubelet/pods/e71b25c5-f3cb-4ed4-b4c3-aa0efae85b6c/volumes"
Sep 18 20:08:40.025502 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:40.025464    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:08:41.642570 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:41.642530    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:08:41.746295 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:41.746259    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:08:43.812924 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:43.812885    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:08:47.538260 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:47.538223    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:08:53.015134 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:53.015092    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:08:53.116464 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:53.116430    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:08:56.208753 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:56.208715    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-c2sqn"]
Sep 18 20:08:56.209174 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:56.209111    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e71b25c5-f3cb-4ed4-b4c3-aa0efae85b6c" containerName="main"
Sep 18 20:08:56.209174 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:56.209124    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e71b25c5-f3cb-4ed4-b4c3-aa0efae85b6c" containerName="main"
Sep 18 20:08:56.209174 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:56.209140    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b2011e8-a73f-43bf-a9ea-fdd27402ac08" containerName="main"
Sep 18 20:08:56.209174 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:56.209146    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b2011e8-a73f-43bf-a9ea-fdd27402ac08" containerName="main"
Sep 18 20:08:56.209174 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:56.209169    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db4f7fec-dab0-44ba-952b-6c757f76024f" containerName="main"
Sep 18 20:08:56.209174 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:56.209175    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="db4f7fec-dab0-44ba-952b-6c757f76024f" containerName="main"
Sep 18 20:08:56.209377 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:56.209231    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e71b25c5-f3cb-4ed4-b4c3-aa0efae85b6c" containerName="main"
Sep 18 20:08:56.209377 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:56.209238    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b2011e8-a73f-43bf-a9ea-fdd27402ac08" containerName="main"
Sep 18 20:08:56.209377 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:56.209246    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="db4f7fec-dab0-44ba-952b-6c757f76024f" containerName="main"
Sep 18 20:08:56.213813 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:56.213792    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-c2sqn"
Sep 18 20:08:56.216367 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:56.216336    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"kube-root-ca.crt\""
Sep 18 20:08:56.217360 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:56.217335    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"test-div-v1-kserve-self-signed-certs\""
Sep 18 20:08:56.217515 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:56.217376    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"openshift-service-ca.crt\""
Sep 18 20:08:56.217515 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:56.217376    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"default-dockercfg-gkvz5\""
Sep 18 20:08:56.221191 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:56.221157    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-c2sqn"]
Sep 18 20:08:56.231199 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:56.231162    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:08:56.305376 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:56.305323    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebf1ae98-682e-48ca-95a3-81d8be612ff1-home\") pod \"test-div-v1-kserve-fb566d667-c2sqn\" (UID: \"ebf1ae98-682e-48ca-95a3-81d8be612ff1\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-c2sqn"
Sep 18 20:08:56.305565 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:56.305391    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9ngg\" (UniqueName: \"kubernetes.io/projected/ebf1ae98-682e-48ca-95a3-81d8be612ff1-kube-api-access-c9ngg\") pod \"test-div-v1-kserve-fb566d667-c2sqn\" (UID: \"ebf1ae98-682e-48ca-95a3-81d8be612ff1\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-c2sqn"
Sep 18 20:08:56.305565 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:56.305437    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebf1ae98-682e-48ca-95a3-81d8be612ff1-model-cache\") pod \"test-div-v1-kserve-fb566d667-c2sqn\" (UID: \"ebf1ae98-682e-48ca-95a3-81d8be612ff1\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-c2sqn"
Sep 18 20:08:56.305565 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:56.305464    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebf1ae98-682e-48ca-95a3-81d8be612ff1-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-c2sqn\" (UID: \"ebf1ae98-682e-48ca-95a3-81d8be612ff1\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-c2sqn"
Sep 18 20:08:56.305565 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:56.305486    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebf1ae98-682e-48ca-95a3-81d8be612ff1-dshm\") pod \"test-div-v1-kserve-fb566d667-c2sqn\" (UID: \"ebf1ae98-682e-48ca-95a3-81d8be612ff1\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-c2sqn"
Sep 18 20:08:56.305565 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:56.305506    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebf1ae98-682e-48ca-95a3-81d8be612ff1-tls-certs\") pod \"test-div-v1-kserve-fb566d667-c2sqn\" (UID: \"ebf1ae98-682e-48ca-95a3-81d8be612ff1\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-c2sqn"
Sep 18 20:08:56.406782 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:56.406739    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9ngg\" (UniqueName: \"kubernetes.io/projected/ebf1ae98-682e-48ca-95a3-81d8be612ff1-kube-api-access-c9ngg\") pod \"test-div-v1-kserve-fb566d667-c2sqn\" (UID: \"ebf1ae98-682e-48ca-95a3-81d8be612ff1\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-c2sqn"
Sep 18 20:08:56.406972 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:56.406799    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebf1ae98-682e-48ca-95a3-81d8be612ff1-model-cache\") pod \"test-div-v1-kserve-fb566d667-c2sqn\" (UID: \"ebf1ae98-682e-48ca-95a3-81d8be612ff1\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-c2sqn"
Sep 18 20:08:56.406972 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:56.406828    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebf1ae98-682e-48ca-95a3-81d8be612ff1-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-c2sqn\" (UID: \"ebf1ae98-682e-48ca-95a3-81d8be612ff1\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-c2sqn"
Sep 18 20:08:56.406972 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:56.406852    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebf1ae98-682e-48ca-95a3-81d8be612ff1-dshm\") pod \"test-div-v1-kserve-fb566d667-c2sqn\" (UID: \"ebf1ae98-682e-48ca-95a3-81d8be612ff1\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-c2sqn"
Sep 18 20:08:56.406972 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:56.406866    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebf1ae98-682e-48ca-95a3-81d8be612ff1-tls-certs\") pod \"test-div-v1-kserve-fb566d667-c2sqn\" (UID: \"ebf1ae98-682e-48ca-95a3-81d8be612ff1\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-c2sqn"
Sep 18 20:08:56.406972 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:56.406895    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebf1ae98-682e-48ca-95a3-81d8be612ff1-home\") pod \"test-div-v1-kserve-fb566d667-c2sqn\" (UID: \"ebf1ae98-682e-48ca-95a3-81d8be612ff1\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-c2sqn"
Sep 18 20:08:56.407419 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:56.407396    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebf1ae98-682e-48ca-95a3-81d8be612ff1-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-c2sqn\" (UID: \"ebf1ae98-682e-48ca-95a3-81d8be612ff1\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-c2sqn"
Sep 18 20:08:56.407490 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:56.407419    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebf1ae98-682e-48ca-95a3-81d8be612ff1-model-cache\") pod \"test-div-v1-kserve-fb566d667-c2sqn\" (UID: \"ebf1ae98-682e-48ca-95a3-81d8be612ff1\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-c2sqn"
Sep 18 20:08:56.407490 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:56.407434    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebf1ae98-682e-48ca-95a3-81d8be612ff1-home\") pod \"test-div-v1-kserve-fb566d667-c2sqn\" (UID: \"ebf1ae98-682e-48ca-95a3-81d8be612ff1\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-c2sqn"
Sep 18 20:08:56.409299 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:56.409276    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebf1ae98-682e-48ca-95a3-81d8be612ff1-dshm\") pod \"test-div-v1-kserve-fb566d667-c2sqn\" (UID: \"ebf1ae98-682e-48ca-95a3-81d8be612ff1\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-c2sqn"
Sep 18 20:08:56.409472 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:56.409452    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebf1ae98-682e-48ca-95a3-81d8be612ff1-tls-certs\") pod \"test-div-v1-kserve-fb566d667-c2sqn\" (UID: \"ebf1ae98-682e-48ca-95a3-81d8be612ff1\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-c2sqn"
Sep 18 20:08:56.418538 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:56.418507    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9ngg\" (UniqueName: \"kubernetes.io/projected/ebf1ae98-682e-48ca-95a3-81d8be612ff1-kube-api-access-c9ngg\") pod \"test-div-v1-kserve-fb566d667-c2sqn\" (UID: \"ebf1ae98-682e-48ca-95a3-81d8be612ff1\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-c2sqn"
Sep 18 20:08:56.527757 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:56.527655    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-c2sqn"
Sep 18 20:08:56.809601 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:56.809574    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-c2sqn"]
Sep 18 20:08:56.812235 ip-10-0-128-161 kubenswrapper[2564]: W0918 20:08:56.812202    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebf1ae98_682e_48ca_95a3_81d8be612ff1.slice/crio-74ee64a1a26d35eec7e7ccc7942189ad6aebb50cdbeb1726a11a74cbe038783a WatchSource:0}: Error finding container 74ee64a1a26d35eec7e7ccc7942189ad6aebb50cdbeb1726a11a74cbe038783a: Status 404 returned error can't find the container with id 74ee64a1a26d35eec7e7ccc7942189ad6aebb50cdbeb1726a11a74cbe038783a
Sep 18 20:08:57.164777 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:57.164738    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-c2sqn" event={"ID":"ebf1ae98-682e-48ca-95a3-81d8be612ff1","Type":"ContainerStarted","Data":"a0338c78de36b01bd176b42ae460a3686409f5fc4539ea7349f1e0396a904c7e"}
Sep 18 20:08:57.164965 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:57.164793    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-c2sqn" event={"ID":"ebf1ae98-682e-48ca-95a3-81d8be612ff1","Type":"ContainerStarted","Data":"74ee64a1a26d35eec7e7ccc7942189ad6aebb50cdbeb1726a11a74cbe038783a"}
Sep 18 20:08:57.181156 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:57.181097    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-c2sqn" podStartSLOduration=1.181078676 podStartE2EDuration="1.181078676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:08:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:08:57.180062266 +0000 UTC m=+1853.690418357" watchObservedRunningTime="2026-09-18 20:08:57.181078676 +0000 UTC m=+1853.691434766"
Sep 18 20:08:57.528217 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:08:57.528130    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-c2sqn"
Sep 18 20:09:06.528645 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:09:06.528606    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-c2sqn"
Sep 18 20:09:06.541418 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:09:06.541385    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-c2sqn"
Sep 18 20:09:07.224903 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:09:07.224871    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-c2sqn"
Sep 18 20:09:14.085429 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:09:14.085389    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-cs5zk"]
Sep 18 20:09:14.090191 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:09:14.090162    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-cs5zk"
Sep 18 20:09:14.092182 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:09:14.092162    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"test-div-v2-kserve-self-signed-certs\""
Sep 18 20:09:14.096632 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:09:14.096607    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-cs5zk"]
Sep 18 20:09:14.115268 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:09:14.115231    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:09:14.171251 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:09:14.171211    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e26b9d7c-245e-4e59-87a5-3a3df92ed231-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-cs5zk\" (UID: \"e26b9d7c-245e-4e59-87a5-3a3df92ed231\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-cs5zk"
Sep 18 20:09:14.171251 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:09:14.171259    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e26b9d7c-245e-4e59-87a5-3a3df92ed231-home\") pod \"test-div-v2-kserve-5f6fdb6b77-cs5zk\" (UID: \"e26b9d7c-245e-4e59-87a5-3a3df92ed231\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-cs5zk"
Sep 18 20:09:14.171524 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:09:14.171342    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl6n7\" (UniqueName: \"kubernetes.io/projected/e26b9d7c-245e-4e59-87a5-3a3df92ed231-kube-api-access-xl6n7\") pod \"test-div-v2-kserve-5f6fdb6b77-cs5zk\" (UID: \"e26b9d7c-245e-4e59-87a5-3a3df92ed231\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-cs5zk"
Sep 18 20:09:14.171524 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:09:14.171383    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e26b9d7c-245e-4e59-87a5-3a3df92ed231-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-cs5zk\" (UID: \"e26b9d7c-245e-4e59-87a5-3a3df92ed231\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-cs5zk"
Sep 18 20:09:14.171524 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:09:14.171406    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e26b9d7c-245e-4e59-87a5-3a3df92ed231-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-cs5zk\" (UID: \"e26b9d7c-245e-4e59-87a5-3a3df92ed231\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-cs5zk"
Sep 18 20:09:14.171524 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:09:14.171430    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e26b9d7c-245e-4e59-87a5-3a3df92ed231-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-cs5zk\" (UID: \"e26b9d7c-245e-4e59-87a5-3a3df92ed231\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-cs5zk"
Sep 18 20:09:14.272396 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:09:14.272361    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e26b9d7c-245e-4e59-87a5-3a3df92ed231-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-cs5zk\" (UID: \"e26b9d7c-245e-4e59-87a5-3a3df92ed231\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-cs5zk"
Sep 18 20:09:14.272396 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:09:14.272397    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e26b9d7c-245e-4e59-87a5-3a3df92ed231-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-cs5zk\" (UID: \"e26b9d7c-245e-4e59-87a5-3a3df92ed231\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-cs5zk"
Sep 18 20:09:14.272642 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:09:14.272416    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e26b9d7c-245e-4e59-87a5-3a3df92ed231-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-cs5zk\" (UID: \"e26b9d7c-245e-4e59-87a5-3a3df92ed231\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-cs5zk"
Sep 18 20:09:14.272642 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:09:14.272469    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e26b9d7c-245e-4e59-87a5-3a3df92ed231-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-cs5zk\" (UID: \"e26b9d7c-245e-4e59-87a5-3a3df92ed231\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-cs5zk"
Sep 18 20:09:14.272642 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:09:14.272485    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e26b9d7c-245e-4e59-87a5-3a3df92ed231-home\") pod \"test-div-v2-kserve-5f6fdb6b77-cs5zk\" (UID: \"e26b9d7c-245e-4e59-87a5-3a3df92ed231\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-cs5zk"
Sep 18 20:09:14.272642 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:09:14.272548    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl6n7\" (UniqueName: \"kubernetes.io/projected/e26b9d7c-245e-4e59-87a5-3a3df92ed231-kube-api-access-xl6n7\") pod \"test-div-v2-kserve-5f6fdb6b77-cs5zk\" (UID: \"e26b9d7c-245e-4e59-87a5-3a3df92ed231\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-cs5zk"
Sep 18 20:09:14.272937 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:09:14.272907    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e26b9d7c-245e-4e59-87a5-3a3df92ed231-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-cs5zk\" (UID: \"e26b9d7c-245e-4e59-87a5-3a3df92ed231\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-cs5zk"
Sep 18 20:09:14.273044 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:09:14.272941    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e26b9d7c-245e-4e59-87a5-3a3df92ed231-home\") pod \"test-div-v2-kserve-5f6fdb6b77-cs5zk\" (UID: \"e26b9d7c-245e-4e59-87a5-3a3df92ed231\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-cs5zk"
Sep 18 20:09:14.273044 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:09:14.272914    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e26b9d7c-245e-4e59-87a5-3a3df92ed231-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-cs5zk\" (UID: \"e26b9d7c-245e-4e59-87a5-3a3df92ed231\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-cs5zk"
Sep 18 20:09:14.274841 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:09:14.274812    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e26b9d7c-245e-4e59-87a5-3a3df92ed231-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-cs5zk\" (UID: \"e26b9d7c-245e-4e59-87a5-3a3df92ed231\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-cs5zk"
Sep 18 20:09:14.275107 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:09:14.275087    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e26b9d7c-245e-4e59-87a5-3a3df92ed231-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-cs5zk\" (UID: \"e26b9d7c-245e-4e59-87a5-3a3df92ed231\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-cs5zk"
Sep 18 20:09:14.280404 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:09:14.280377    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl6n7\" (UniqueName: \"kubernetes.io/projected/e26b9d7c-245e-4e59-87a5-3a3df92ed231-kube-api-access-xl6n7\") pod \"test-div-v2-kserve-5f6fdb6b77-cs5zk\" (UID: \"e26b9d7c-245e-4e59-87a5-3a3df92ed231\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-cs5zk"
Sep 18 20:09:14.403820 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:09:14.403767    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-cs5zk"
Sep 18 20:09:14.689224 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:09:14.689137    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-cs5zk"]
Sep 18 20:09:14.693147 ip-10-0-128-161 kubenswrapper[2564]: W0918 20:09:14.693115    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode26b9d7c_245e_4e59_87a5_3a3df92ed231.slice/crio-971bead8d93380a82742f6bc630f512318fc60448c82495d804aaedd9a1d07e6 WatchSource:0}: Error finding container 971bead8d93380a82742f6bc630f512318fc60448c82495d804aaedd9a1d07e6: Status 404 returned error can't find the container with id 971bead8d93380a82742f6bc630f512318fc60448c82495d804aaedd9a1d07e6
Sep 18 20:09:15.255533 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:09:15.255501    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-cs5zk" event={"ID":"e26b9d7c-245e-4e59-87a5-3a3df92ed231","Type":"ContainerStarted","Data":"6a342082ed2c7bd5f538be71a001b46fa5f269509538ae83ddffc24b3f669a01"}
Sep 18 20:09:15.255932 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:09:15.255550    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-cs5zk" event={"ID":"e26b9d7c-245e-4e59-87a5-3a3df92ed231","Type":"ContainerStarted","Data":"971bead8d93380a82742f6bc630f512318fc60448c82495d804aaedd9a1d07e6"}
Sep 18 20:09:15.272128 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:09:15.272071    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-cs5zk" podStartSLOduration=1.272051977 podStartE2EDuration="1.272051977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:09:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:09:15.270482868 +0000 UTC m=+1871.780838982" watchObservedRunningTime="2026-09-18 20:09:15.272051977 +0000 UTC m=+1871.782408069"
Sep 18 20:09:15.404164 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:09:15.404124    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-cs5zk"
Sep 18 20:09:20.566781 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:09:20.566728    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:09:21.206892 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:09:21.206860    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:09:21.462874 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:09:21.462788    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69bhj7h"]
Sep 18 20:09:21.466577 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:09:21.466551    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69bhj7h"
Sep 18 20:09:21.469038 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:09:21.469009    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 18 20:09:21.469736 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:09:21.469593    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-ghbfp\""
Sep 18 20:09:21.469736 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:09:21.469595    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 18 20:09:21.470320 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:09:21.470152    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-epp-sa-dockercfg-74d7t\""
Sep 18 20:09:21.470320 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:09:21.470216    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 18 20:09:21.470320 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:09:21.470245    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:09:21.475198 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:09:21.475173    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69bhj7h"]
Sep 18 20:09:21.565637 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:09:21.565598    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:09:21.641372 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:09:21.641326    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34385379-07fe-4623-a216-f7b04275adad-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69bhj7h\" (UID: \"34385379-07fe-4623-a216-f7b04275adad\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69bhj7h"
Sep 18 20:09:21.641822 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:09:21.641461    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl44v\" (UniqueName: \"kubernetes.io/projected/34385379-07fe-4623-a216-f7b04275adad-kube-api-access-fl44v\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69bhj7h\" (UID: \"34385379-07fe-4623-a216-f7b04275adad\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69bhj7h"
Sep 18 20:09:21.742304 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:09:21.742211    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34385379-07fe-4623-a216-f7b04275adad-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69bhj7h\" (UID: \"34385379-07fe-4623-a216-f7b04275adad\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69bhj7h"
Sep 18 20:09:21.742470 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:09:21.742310    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl44v\" (UniqueName: \"kubernetes.io/projected/34385379-07fe-4623-a216-f7b04275adad-kube-api-access-fl44v\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69bhj7h\" (UID: \"34385379-07fe-4623-a216-f7b04275adad\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69bhj7h"
Sep 18 20:09:21.744973 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:09:21.744945    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34385379-07fe-4623-a216-f7b04275adad-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69bhj7h\" (UID: \"34385379-07fe-4623-a216-f7b04275adad\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69bhj7h"
Sep 18 20:09:21.753480 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:09:21.753447    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl44v\" (UniqueName: \"kubernetes.io/projected/34385379-07fe-4623-a216-f7b04275adad-kube-api-access-fl44v\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69bhj7h\" (UID: \"34385379-07fe-4623-a216-f7b04275adad\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69bhj7h"
Sep 18 20:09:21.780920 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:09:21.780882    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69bhj7h"
Sep 18 20:09:22.058044 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:09:22.058018    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69bhj7h"]
Sep 18 20:09:22.060579 ip-10-0-128-161 kubenswrapper[2564]: W0918 20:09:22.060546    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34385379_07fe_4623_a216_f7b04275adad.slice/crio-7e167f3136989c52d0c84daf4bf4b860d9d80ba521c6578ff77533de8478d89c WatchSource:0}: Error finding container 7e167f3136989c52d0c84daf4bf4b860d9d80ba521c6578ff77533de8478d89c: Status 404 returned error can't find the container with id 7e167f3136989c52d0c84daf4bf4b860d9d80ba521c6578ff77533de8478d89c
Sep 18 20:09:22.292826 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:09:22.292791    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69bhj7h" event={"ID":"34385379-07fe-4623-a216-f7b04275adad","Type":"ContainerStarted","Data":"662533695d1af28970081cbac72b577f4c75f35da8cc9fe69a0b816353a06cb0"}
Sep 18 20:09:22.293069 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:09:22.292850    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69bhj7h" event={"ID":"34385379-07fe-4623-a216-f7b04275adad","Type":"ContainerStarted","Data":"7e167f3136989c52d0c84daf4bf4b860d9d80ba521c6578ff77533de8478d89c"}
Sep 18 20:09:22.293069 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:09:22.292876    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69bhj7h"
Sep 18 20:09:22.310133 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:09:22.310031    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69bhj7h" podStartSLOduration=1.310009286 podStartE2EDuration="1.310009286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:09:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:09:22.308735138 +0000 UTC m=+1878.819091235" watchObservedRunningTime="2026-09-18 20:09:22.310009286 +0000 UTC m=+1878.820365370"
Sep 18 20:09:24.404464 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:09:24.404414    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-cs5zk"
Sep 18 20:09:24.416945 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:09:24.416920    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-cs5zk"
Sep 18 20:09:25.319586 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:09:25.319551    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-cs5zk"
Sep 18 20:09:26.110105 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:09:26.110006    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:09:26.223339 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:09:26.223292    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:09:53.300112 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:09:53.300083    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69bhj7h"
Sep 18 20:10:01.368128 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:01.368086    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:10:09.706941 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:09.706901    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:10:09.805851 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:09.805804    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:10:09.911872 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:09.911840    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:10:10.414531 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:10.414489    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:10:10.517140 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:10.517101    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:10:10.616355 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:10.616320    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:10:10.867477 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:10.867397    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-cs5zk"]
Sep 18 20:10:10.876371 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:10.876340    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:10:11.685823 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:11.685775    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-c2sqn"]
Sep 18 20:10:11.705955 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:11.705926    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:10:14.779671 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:14.779631    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:10:16.195285 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:16.195247    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:10:21.832267 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:21.832224    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:10:22.074873 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:22.074838    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:10:22.175788 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:22.175743    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:10:25.887141 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:25.887101    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-cs5zk" podUID="e26b9d7c-245e-4e59-87a5-3a3df92ed231" containerName="main" containerID="cri-o://6a342082ed2c7bd5f538be71a001b46fa5f269509538ae83ddffc24b3f669a01" gracePeriod=45
Sep 18 20:10:26.150815 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:26.150747    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-cs5zk"
Sep 18 20:10:26.241064 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:26.241027    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e26b9d7c-245e-4e59-87a5-3a3df92ed231-model-cache\") pod \"e26b9d7c-245e-4e59-87a5-3a3df92ed231\" (UID: \"e26b9d7c-245e-4e59-87a5-3a3df92ed231\") "
Sep 18 20:10:26.241246 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:26.241138    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xl6n7\" (UniqueName: \"kubernetes.io/projected/e26b9d7c-245e-4e59-87a5-3a3df92ed231-kube-api-access-xl6n7\") pod \"e26b9d7c-245e-4e59-87a5-3a3df92ed231\" (UID: \"e26b9d7c-245e-4e59-87a5-3a3df92ed231\") "
Sep 18 20:10:26.241246 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:26.241172    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e26b9d7c-245e-4e59-87a5-3a3df92ed231-tmp-dir\") pod \"e26b9d7c-245e-4e59-87a5-3a3df92ed231\" (UID: \"e26b9d7c-245e-4e59-87a5-3a3df92ed231\") "
Sep 18 20:10:26.241246 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:26.241214    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e26b9d7c-245e-4e59-87a5-3a3df92ed231-home\") pod \"e26b9d7c-245e-4e59-87a5-3a3df92ed231\" (UID: \"e26b9d7c-245e-4e59-87a5-3a3df92ed231\") "
Sep 18 20:10:26.241404 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:26.241341    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e26b9d7c-245e-4e59-87a5-3a3df92ed231-model-cache" (OuterVolumeSpecName: "model-cache") pod "e26b9d7c-245e-4e59-87a5-3a3df92ed231" (UID: "e26b9d7c-245e-4e59-87a5-3a3df92ed231"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:10:26.241459 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:26.241408    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e26b9d7c-245e-4e59-87a5-3a3df92ed231-tls-certs\") pod \"e26b9d7c-245e-4e59-87a5-3a3df92ed231\" (UID: \"e26b9d7c-245e-4e59-87a5-3a3df92ed231\") "
Sep 18 20:10:26.241531 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:26.241486    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e26b9d7c-245e-4e59-87a5-3a3df92ed231-dshm\") pod \"e26b9d7c-245e-4e59-87a5-3a3df92ed231\" (UID: \"e26b9d7c-245e-4e59-87a5-3a3df92ed231\") "
Sep 18 20:10:26.241531 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:26.241411    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e26b9d7c-245e-4e59-87a5-3a3df92ed231-home" (OuterVolumeSpecName: "home") pod "e26b9d7c-245e-4e59-87a5-3a3df92ed231" (UID: "e26b9d7c-245e-4e59-87a5-3a3df92ed231"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:10:26.241637 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:26.241427    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e26b9d7c-245e-4e59-87a5-3a3df92ed231-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e26b9d7c-245e-4e59-87a5-3a3df92ed231" (UID: "e26b9d7c-245e-4e59-87a5-3a3df92ed231"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:10:26.241823 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:26.241802    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e26b9d7c-245e-4e59-87a5-3a3df92ed231-tmp-dir\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:10:26.241890 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:26.241828    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e26b9d7c-245e-4e59-87a5-3a3df92ed231-home\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:10:26.241890 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:26.241841    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e26b9d7c-245e-4e59-87a5-3a3df92ed231-model-cache\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:10:26.243852 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:26.243819    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e26b9d7c-245e-4e59-87a5-3a3df92ed231-dshm" (OuterVolumeSpecName: "dshm") pod "e26b9d7c-245e-4e59-87a5-3a3df92ed231" (UID: "e26b9d7c-245e-4e59-87a5-3a3df92ed231"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:10:26.243963 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:26.243850    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e26b9d7c-245e-4e59-87a5-3a3df92ed231-kube-api-access-xl6n7" (OuterVolumeSpecName: "kube-api-access-xl6n7") pod "e26b9d7c-245e-4e59-87a5-3a3df92ed231" (UID: "e26b9d7c-245e-4e59-87a5-3a3df92ed231"). InnerVolumeSpecName "kube-api-access-xl6n7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:10:26.243963 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:26.243923    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e26b9d7c-245e-4e59-87a5-3a3df92ed231-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e26b9d7c-245e-4e59-87a5-3a3df92ed231" (UID: "e26b9d7c-245e-4e59-87a5-3a3df92ed231"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:10:26.342667 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:26.342628    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xl6n7\" (UniqueName: \"kubernetes.io/projected/e26b9d7c-245e-4e59-87a5-3a3df92ed231-kube-api-access-xl6n7\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:10:26.342667 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:26.342665    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e26b9d7c-245e-4e59-87a5-3a3df92ed231-tls-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:10:26.342878 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:26.342680    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e26b9d7c-245e-4e59-87a5-3a3df92ed231-dshm\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:10:26.626276 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:26.626180    2564 generic.go:356] "Generic (PLEG): container finished" podID="e26b9d7c-245e-4e59-87a5-3a3df92ed231" containerID="6a342082ed2c7bd5f538be71a001b46fa5f269509538ae83ddffc24b3f669a01" exitCode=0
Sep 18 20:10:26.626276 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:26.626231    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-cs5zk"
Sep 18 20:10:26.626479 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:26.626269    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-cs5zk" event={"ID":"e26b9d7c-245e-4e59-87a5-3a3df92ed231","Type":"ContainerDied","Data":"6a342082ed2c7bd5f538be71a001b46fa5f269509538ae83ddffc24b3f669a01"}
Sep 18 20:10:26.626479 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:26.626347    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-cs5zk" event={"ID":"e26b9d7c-245e-4e59-87a5-3a3df92ed231","Type":"ContainerDied","Data":"971bead8d93380a82742f6bc630f512318fc60448c82495d804aaedd9a1d07e6"}
Sep 18 20:10:26.626479 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:26.626370    2564 scope.go:117] "RemoveContainer" containerID="6a342082ed2c7bd5f538be71a001b46fa5f269509538ae83ddffc24b3f669a01"
Sep 18 20:10:26.636811 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:26.636789    2564 scope.go:117] "RemoveContainer" containerID="6a342082ed2c7bd5f538be71a001b46fa5f269509538ae83ddffc24b3f669a01"
Sep 18 20:10:26.637196 ip-10-0-128-161 kubenswrapper[2564]: E0918 20:10:26.637174    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a342082ed2c7bd5f538be71a001b46fa5f269509538ae83ddffc24b3f669a01\": container with ID starting with 6a342082ed2c7bd5f538be71a001b46fa5f269509538ae83ddffc24b3f669a01 not found: ID does not exist" containerID="6a342082ed2c7bd5f538be71a001b46fa5f269509538ae83ddffc24b3f669a01"
Sep 18 20:10:26.637284 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:26.637210    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a342082ed2c7bd5f538be71a001b46fa5f269509538ae83ddffc24b3f669a01"} err="failed to get container status \"6a342082ed2c7bd5f538be71a001b46fa5f269509538ae83ddffc24b3f669a01\": rpc error: code = NotFound desc = could not find container \"6a342082ed2c7bd5f538be71a001b46fa5f269509538ae83ddffc24b3f669a01\": container with ID starting with 6a342082ed2c7bd5f538be71a001b46fa5f269509538ae83ddffc24b3f669a01 not found: ID does not exist"
Sep 18 20:10:26.649866 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:26.649830    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-cs5zk"]
Sep 18 20:10:26.651618 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:26.651590    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-cs5zk"]
Sep 18 20:10:26.706503 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:26.706468    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-c2sqn" podUID="ebf1ae98-682e-48ca-95a3-81d8be612ff1" containerName="main" containerID="cri-o://a0338c78de36b01bd176b42ae460a3686409f5fc4539ea7349f1e0396a904c7e" gracePeriod=45
Sep 18 20:10:26.966033 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:26.965975    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-c2sqn"
Sep 18 20:10:27.050313 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:27.050277    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebf1ae98-682e-48ca-95a3-81d8be612ff1-tmp-dir\") pod \"ebf1ae98-682e-48ca-95a3-81d8be612ff1\" (UID: \"ebf1ae98-682e-48ca-95a3-81d8be612ff1\") "
Sep 18 20:10:27.050313 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:27.050322    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebf1ae98-682e-48ca-95a3-81d8be612ff1-home\") pod \"ebf1ae98-682e-48ca-95a3-81d8be612ff1\" (UID: \"ebf1ae98-682e-48ca-95a3-81d8be612ff1\") "
Sep 18 20:10:27.050575 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:27.050343    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebf1ae98-682e-48ca-95a3-81d8be612ff1-tls-certs\") pod \"ebf1ae98-682e-48ca-95a3-81d8be612ff1\" (UID: \"ebf1ae98-682e-48ca-95a3-81d8be612ff1\") "
Sep 18 20:10:27.050575 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:27.050369    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c9ngg\" (UniqueName: \"kubernetes.io/projected/ebf1ae98-682e-48ca-95a3-81d8be612ff1-kube-api-access-c9ngg\") pod \"ebf1ae98-682e-48ca-95a3-81d8be612ff1\" (UID: \"ebf1ae98-682e-48ca-95a3-81d8be612ff1\") "
Sep 18 20:10:27.050575 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:27.050438    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebf1ae98-682e-48ca-95a3-81d8be612ff1-dshm\") pod \"ebf1ae98-682e-48ca-95a3-81d8be612ff1\" (UID: \"ebf1ae98-682e-48ca-95a3-81d8be612ff1\") "
Sep 18 20:10:27.050575 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:27.050493    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebf1ae98-682e-48ca-95a3-81d8be612ff1-model-cache\") pod \"ebf1ae98-682e-48ca-95a3-81d8be612ff1\" (UID: \"ebf1ae98-682e-48ca-95a3-81d8be612ff1\") "
Sep 18 20:10:27.050788 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:27.050629    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebf1ae98-682e-48ca-95a3-81d8be612ff1-home" (OuterVolumeSpecName: "home") pod "ebf1ae98-682e-48ca-95a3-81d8be612ff1" (UID: "ebf1ae98-682e-48ca-95a3-81d8be612ff1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:10:27.050788 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:27.050643    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebf1ae98-682e-48ca-95a3-81d8be612ff1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ebf1ae98-682e-48ca-95a3-81d8be612ff1" (UID: "ebf1ae98-682e-48ca-95a3-81d8be612ff1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:10:27.050900 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:27.050837    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebf1ae98-682e-48ca-95a3-81d8be612ff1-tmp-dir\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:10:27.050900 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:27.050855    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebf1ae98-682e-48ca-95a3-81d8be612ff1-home\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:10:27.050900 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:27.050853    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebf1ae98-682e-48ca-95a3-81d8be612ff1-model-cache" (OuterVolumeSpecName: "model-cache") pod "ebf1ae98-682e-48ca-95a3-81d8be612ff1" (UID: "ebf1ae98-682e-48ca-95a3-81d8be612ff1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:10:27.052662 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:27.052640    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebf1ae98-682e-48ca-95a3-81d8be612ff1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ebf1ae98-682e-48ca-95a3-81d8be612ff1" (UID: "ebf1ae98-682e-48ca-95a3-81d8be612ff1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:10:27.052791 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:27.052676    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebf1ae98-682e-48ca-95a3-81d8be612ff1-kube-api-access-c9ngg" (OuterVolumeSpecName: "kube-api-access-c9ngg") pod "ebf1ae98-682e-48ca-95a3-81d8be612ff1" (UID: "ebf1ae98-682e-48ca-95a3-81d8be612ff1"). InnerVolumeSpecName "kube-api-access-c9ngg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:10:27.052791 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:27.052739    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebf1ae98-682e-48ca-95a3-81d8be612ff1-dshm" (OuterVolumeSpecName: "dshm") pod "ebf1ae98-682e-48ca-95a3-81d8be612ff1" (UID: "ebf1ae98-682e-48ca-95a3-81d8be612ff1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:10:27.151516 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:27.151476    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebf1ae98-682e-48ca-95a3-81d8be612ff1-model-cache\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:10:27.151516 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:27.151512    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebf1ae98-682e-48ca-95a3-81d8be612ff1-tls-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:10:27.151730 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:27.151526    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9ngg\" (UniqueName: \"kubernetes.io/projected/ebf1ae98-682e-48ca-95a3-81d8be612ff1-kube-api-access-c9ngg\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:10:27.151730 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:27.151539    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebf1ae98-682e-48ca-95a3-81d8be612ff1-dshm\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:10:27.639588 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:27.639554    2564 generic.go:356] "Generic (PLEG): container finished" podID="ebf1ae98-682e-48ca-95a3-81d8be612ff1" containerID="a0338c78de36b01bd176b42ae460a3686409f5fc4539ea7349f1e0396a904c7e" exitCode=0
Sep 18 20:10:27.639810 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:27.639614    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-c2sqn"
Sep 18 20:10:27.639810 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:27.639638    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-c2sqn" event={"ID":"ebf1ae98-682e-48ca-95a3-81d8be612ff1","Type":"ContainerDied","Data":"a0338c78de36b01bd176b42ae460a3686409f5fc4539ea7349f1e0396a904c7e"}
Sep 18 20:10:27.639810 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:27.639717    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-c2sqn" event={"ID":"ebf1ae98-682e-48ca-95a3-81d8be612ff1","Type":"ContainerDied","Data":"74ee64a1a26d35eec7e7ccc7942189ad6aebb50cdbeb1726a11a74cbe038783a"}
Sep 18 20:10:27.639810 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:27.639740    2564 scope.go:117] "RemoveContainer" containerID="a0338c78de36b01bd176b42ae460a3686409f5fc4539ea7349f1e0396a904c7e"
Sep 18 20:10:27.649364 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:27.649341    2564 scope.go:117] "RemoveContainer" containerID="a0338c78de36b01bd176b42ae460a3686409f5fc4539ea7349f1e0396a904c7e"
Sep 18 20:10:27.649710 ip-10-0-128-161 kubenswrapper[2564]: E0918 20:10:27.649692    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0338c78de36b01bd176b42ae460a3686409f5fc4539ea7349f1e0396a904c7e\": container with ID starting with a0338c78de36b01bd176b42ae460a3686409f5fc4539ea7349f1e0396a904c7e not found: ID does not exist" containerID="a0338c78de36b01bd176b42ae460a3686409f5fc4539ea7349f1e0396a904c7e"
Sep 18 20:10:27.649770 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:27.649724    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0338c78de36b01bd176b42ae460a3686409f5fc4539ea7349f1e0396a904c7e"} err="failed to get container status \"a0338c78de36b01bd176b42ae460a3686409f5fc4539ea7349f1e0396a904c7e\": rpc error: code = NotFound desc = could not find container \"a0338c78de36b01bd176b42ae460a3686409f5fc4539ea7349f1e0396a904c7e\": container with ID starting with a0338c78de36b01bd176b42ae460a3686409f5fc4539ea7349f1e0396a904c7e not found: ID does not exist"
Sep 18 20:10:27.661581 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:27.661548    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-c2sqn"]
Sep 18 20:10:27.664634 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:27.664610    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-c2sqn"]
Sep 18 20:10:28.000479 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:28.000374    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e26b9d7c-245e-4e59-87a5-3a3df92ed231" path="/var/lib/kubelet/pods/e26b9d7c-245e-4e59-87a5-3a3df92ed231/volumes"
Sep 18 20:10:28.001015 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:28.000976    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebf1ae98-682e-48ca-95a3-81d8be612ff1" path="/var/lib/kubelet/pods/ebf1ae98-682e-48ca-95a3-81d8be612ff1/volumes"
Sep 18 20:10:43.603431 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:43.603387    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpcc6"]
Sep 18 20:10:43.605800 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:43.603770    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e26b9d7c-245e-4e59-87a5-3a3df92ed231" containerName="main"
Sep 18 20:10:43.605800 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:43.603787    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e26b9d7c-245e-4e59-87a5-3a3df92ed231" containerName="main"
Sep 18 20:10:43.605800 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:43.603814    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebf1ae98-682e-48ca-95a3-81d8be612ff1" containerName="main"
Sep 18 20:10:43.605800 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:43.603824    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebf1ae98-682e-48ca-95a3-81d8be612ff1" containerName="main"
Sep 18 20:10:43.605800 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:43.603928    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e26b9d7c-245e-4e59-87a5-3a3df92ed231" containerName="main"
Sep 18 20:10:43.605800 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:43.603969    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebf1ae98-682e-48ca-95a3-81d8be612ff1" containerName="main"
Sep 18 20:10:43.606893 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:43.606875    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpcc6"
Sep 18 20:10:43.609675 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:43.609646    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"openshift-service-ca.crt\""
Sep 18 20:10:43.609869 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:43.609746    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"default-dockercfg-btwqd\""
Sep 18 20:10:43.609869 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:43.609764    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"stop-v2-kserve-self-signed-certs\""
Sep 18 20:10:43.609869 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:43.609846    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"kube-root-ca.crt\""
Sep 18 20:10:43.616193 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:43.616166    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpcc6"]
Sep 18 20:10:43.704284 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:43.704246    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:10:43.705265 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:43.705240    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf7k4\" (UniqueName: \"kubernetes.io/projected/f67ffaea-1af3-455c-bc84-174dc116a709-kube-api-access-tf7k4\") pod \"stop-v2-kserve-7f68747c45-cpcc6\" (UID: \"f67ffaea-1af3-455c-bc84-174dc116a709\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpcc6"
Sep 18 20:10:43.705350 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:43.705283    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f67ffaea-1af3-455c-bc84-174dc116a709-model-cache\") pod \"stop-v2-kserve-7f68747c45-cpcc6\" (UID: \"f67ffaea-1af3-455c-bc84-174dc116a709\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpcc6"
Sep 18 20:10:43.705350 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:43.705302    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f67ffaea-1af3-455c-bc84-174dc116a709-tmp-dir\") pod \"stop-v2-kserve-7f68747c45-cpcc6\" (UID: \"f67ffaea-1af3-455c-bc84-174dc116a709\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpcc6"
Sep 18 20:10:43.705433 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:43.705374    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f67ffaea-1af3-455c-bc84-174dc116a709-tls-certs\") pod \"stop-v2-kserve-7f68747c45-cpcc6\" (UID: \"f67ffaea-1af3-455c-bc84-174dc116a709\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpcc6"
Sep 18 20:10:43.705433 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:43.705422    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f67ffaea-1af3-455c-bc84-174dc116a709-home\") pod \"stop-v2-kserve-7f68747c45-cpcc6\" (UID: \"f67ffaea-1af3-455c-bc84-174dc116a709\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpcc6"
Sep 18 20:10:43.705513 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:43.705438    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f67ffaea-1af3-455c-bc84-174dc116a709-dshm\") pod \"stop-v2-kserve-7f68747c45-cpcc6\" (UID: \"f67ffaea-1af3-455c-bc84-174dc116a709\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpcc6"
Sep 18 20:10:43.806172 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:43.806123    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f67ffaea-1af3-455c-bc84-174dc116a709-tls-certs\") pod \"stop-v2-kserve-7f68747c45-cpcc6\" (UID: \"f67ffaea-1af3-455c-bc84-174dc116a709\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpcc6"
Sep 18 20:10:43.806384 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:43.806208    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f67ffaea-1af3-455c-bc84-174dc116a709-home\") pod \"stop-v2-kserve-7f68747c45-cpcc6\" (UID: \"f67ffaea-1af3-455c-bc84-174dc116a709\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpcc6"
Sep 18 20:10:43.806384 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:43.806224    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f67ffaea-1af3-455c-bc84-174dc116a709-dshm\") pod \"stop-v2-kserve-7f68747c45-cpcc6\" (UID: \"f67ffaea-1af3-455c-bc84-174dc116a709\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpcc6"
Sep 18 20:10:43.806384 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:43.806248    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf7k4\" (UniqueName: \"kubernetes.io/projected/f67ffaea-1af3-455c-bc84-174dc116a709-kube-api-access-tf7k4\") pod \"stop-v2-kserve-7f68747c45-cpcc6\" (UID: \"f67ffaea-1af3-455c-bc84-174dc116a709\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpcc6"
Sep 18 20:10:43.806384 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:43.806276    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f67ffaea-1af3-455c-bc84-174dc116a709-model-cache\") pod \"stop-v2-kserve-7f68747c45-cpcc6\" (UID: \"f67ffaea-1af3-455c-bc84-174dc116a709\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpcc6"
Sep 18 20:10:43.806384 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:43.806306    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f67ffaea-1af3-455c-bc84-174dc116a709-tmp-dir\") pod \"stop-v2-kserve-7f68747c45-cpcc6\" (UID: \"f67ffaea-1af3-455c-bc84-174dc116a709\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpcc6"
Sep 18 20:10:43.806749 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:43.806724    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f67ffaea-1af3-455c-bc84-174dc116a709-home\") pod \"stop-v2-kserve-7f68747c45-cpcc6\" (UID: \"f67ffaea-1af3-455c-bc84-174dc116a709\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpcc6"
Sep 18 20:10:43.806823 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:43.806795    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f67ffaea-1af3-455c-bc84-174dc116a709-tmp-dir\") pod \"stop-v2-kserve-7f68747c45-cpcc6\" (UID: \"f67ffaea-1af3-455c-bc84-174dc116a709\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpcc6"
Sep 18 20:10:43.806885 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:43.806834    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f67ffaea-1af3-455c-bc84-174dc116a709-model-cache\") pod \"stop-v2-kserve-7f68747c45-cpcc6\" (UID: \"f67ffaea-1af3-455c-bc84-174dc116a709\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpcc6"
Sep 18 20:10:43.808658 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:43.808635    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f67ffaea-1af3-455c-bc84-174dc116a709-dshm\") pod \"stop-v2-kserve-7f68747c45-cpcc6\" (UID: \"f67ffaea-1af3-455c-bc84-174dc116a709\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpcc6"
Sep 18 20:10:43.809038 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:43.809021    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f67ffaea-1af3-455c-bc84-174dc116a709-tls-certs\") pod \"stop-v2-kserve-7f68747c45-cpcc6\" (UID: \"f67ffaea-1af3-455c-bc84-174dc116a709\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpcc6"
Sep 18 20:10:43.814705 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:43.814675    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf7k4\" (UniqueName: \"kubernetes.io/projected/f67ffaea-1af3-455c-bc84-174dc116a709-kube-api-access-tf7k4\") pod \"stop-v2-kserve-7f68747c45-cpcc6\" (UID: \"f67ffaea-1af3-455c-bc84-174dc116a709\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpcc6"
Sep 18 20:10:43.918457 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:43.918413    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpcc6"
Sep 18 20:10:44.197969 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:44.197940    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpcc6"]
Sep 18 20:10:44.201365 ip-10-0-128-161 kubenswrapper[2564]: W0918 20:10:44.201321    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf67ffaea_1af3_455c_bc84_174dc116a709.slice/crio-8f9da872c18d78075d2f165df617cdca475310eb2e40d902a4084c56ef8ae955 WatchSource:0}: Error finding container 8f9da872c18d78075d2f165df617cdca475310eb2e40d902a4084c56ef8ae955: Status 404 returned error can't find the container with id 8f9da872c18d78075d2f165df617cdca475310eb2e40d902a4084c56ef8ae955
Sep 18 20:10:44.731119 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:44.731082    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpcc6" event={"ID":"f67ffaea-1af3-455c-bc84-174dc116a709","Type":"ContainerStarted","Data":"11d6df5673d7251548ca2b30317569918d16eaefda36d976acbe8d9f8cecafb4"}
Sep 18 20:10:44.731639 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:44.731165    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpcc6" event={"ID":"f67ffaea-1af3-455c-bc84-174dc116a709","Type":"ContainerStarted","Data":"8f9da872c18d78075d2f165df617cdca475310eb2e40d902a4084c56ef8ae955"}
Sep 18 20:10:44.749904 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:44.749841    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpcc6" podStartSLOduration=1.74981745 podStartE2EDuration="1.74981745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:10:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:10:44.748830717 +0000 UTC m=+1961.259186808" watchObservedRunningTime="2026-09-18 20:10:44.74981745 +0000 UTC m=+1961.260173545"
Sep 18 20:10:44.918765 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:44.918725    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpcc6"
Sep 18 20:10:51.817427 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:51.817392    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:10:53.919166 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:53.919120    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpcc6"
Sep 18 20:10:53.932033 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:53.932000    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpcc6"
Sep 18 20:10:54.793536 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:54.793502    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpcc6"
Sep 18 20:10:57.221529 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:57.221440    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:10:57.275932 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:57.275898    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:10:59.812709 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:59.812660    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:10:59.822873 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:59.822838    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69bhj7h"]
Sep 18 20:10:59.888531 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:10:59.888498    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:03.300385 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:03.300346    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69bhj7h" podUID="34385379-07fe-4623-a216-f7b04275adad" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:11:13.299856 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:13.299816    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69bhj7h" podUID="34385379-07fe-4623-a216-f7b04275adad" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:11:14.846142 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:14.846074    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69bhj7h" podUID="34385379-07fe-4623-a216-f7b04275adad" containerName="main" containerID="cri-o://662533695d1af28970081cbac72b577f4c75f35da8cc9fe69a0b816353a06cb0" gracePeriod=45
Sep 18 20:11:18.004898 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:18.004858    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpcc6"]
Sep 18 20:11:18.032114 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:18.032075    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:23.300068 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:23.300024    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69bhj7h" podUID="34385379-07fe-4623-a216-f7b04275adad" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:11:23.300537 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:23.300138    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69bhj7h"
Sep 18 20:11:33.027692 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:33.027615    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpcc6" podUID="f67ffaea-1af3-455c-bc84-174dc116a709" containerName="main" containerID="cri-o://11d6df5673d7251548ca2b30317569918d16eaefda36d976acbe8d9f8cecafb4" gracePeriod=45
Sep 18 20:11:33.297592 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:33.297559    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpcc6"
Sep 18 20:11:33.299798 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:33.299758    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69bhj7h" podUID="34385379-07fe-4623-a216-f7b04275adad" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:11:33.452304 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:33.452261    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f67ffaea-1af3-455c-bc84-174dc116a709-tmp-dir\") pod \"f67ffaea-1af3-455c-bc84-174dc116a709\" (UID: \"f67ffaea-1af3-455c-bc84-174dc116a709\") "
Sep 18 20:11:33.452486 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:33.452334    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f67ffaea-1af3-455c-bc84-174dc116a709-dshm\") pod \"f67ffaea-1af3-455c-bc84-174dc116a709\" (UID: \"f67ffaea-1af3-455c-bc84-174dc116a709\") "
Sep 18 20:11:33.452486 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:33.452380    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f67ffaea-1af3-455c-bc84-174dc116a709-tls-certs\") pod \"f67ffaea-1af3-455c-bc84-174dc116a709\" (UID: \"f67ffaea-1af3-455c-bc84-174dc116a709\") "
Sep 18 20:11:33.452486 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:33.452406    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f67ffaea-1af3-455c-bc84-174dc116a709-home\") pod \"f67ffaea-1af3-455c-bc84-174dc116a709\" (UID: \"f67ffaea-1af3-455c-bc84-174dc116a709\") "
Sep 18 20:11:33.452486 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:33.452444    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f67ffaea-1af3-455c-bc84-174dc116a709-model-cache\") pod \"f67ffaea-1af3-455c-bc84-174dc116a709\" (UID: \"f67ffaea-1af3-455c-bc84-174dc116a709\") "
Sep 18 20:11:33.452705 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:33.452513    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tf7k4\" (UniqueName: \"kubernetes.io/projected/f67ffaea-1af3-455c-bc84-174dc116a709-kube-api-access-tf7k4\") pod \"f67ffaea-1af3-455c-bc84-174dc116a709\" (UID: \"f67ffaea-1af3-455c-bc84-174dc116a709\") "
Sep 18 20:11:33.452705 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:33.452599    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f67ffaea-1af3-455c-bc84-174dc116a709-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f67ffaea-1af3-455c-bc84-174dc116a709" (UID: "f67ffaea-1af3-455c-bc84-174dc116a709"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:11:33.452816 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:33.452774    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f67ffaea-1af3-455c-bc84-174dc116a709-home" (OuterVolumeSpecName: "home") pod "f67ffaea-1af3-455c-bc84-174dc116a709" (UID: "f67ffaea-1af3-455c-bc84-174dc116a709"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:11:33.452816 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:33.452788    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f67ffaea-1af3-455c-bc84-174dc116a709-model-cache" (OuterVolumeSpecName: "model-cache") pod "f67ffaea-1af3-455c-bc84-174dc116a709" (UID: "f67ffaea-1af3-455c-bc84-174dc116a709"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:11:33.452928 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:33.452905    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f67ffaea-1af3-455c-bc84-174dc116a709-tmp-dir\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:11:33.453017 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:33.452931    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f67ffaea-1af3-455c-bc84-174dc116a709-home\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:11:33.453017 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:33.452945    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f67ffaea-1af3-455c-bc84-174dc116a709-model-cache\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:11:33.454776 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:33.454740    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f67ffaea-1af3-455c-bc84-174dc116a709-dshm" (OuterVolumeSpecName: "dshm") pod "f67ffaea-1af3-455c-bc84-174dc116a709" (UID: "f67ffaea-1af3-455c-bc84-174dc116a709"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:11:33.454906 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:33.454835    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f67ffaea-1af3-455c-bc84-174dc116a709-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f67ffaea-1af3-455c-bc84-174dc116a709" (UID: "f67ffaea-1af3-455c-bc84-174dc116a709"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:11:33.454906 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:33.454830    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f67ffaea-1af3-455c-bc84-174dc116a709-kube-api-access-tf7k4" (OuterVolumeSpecName: "kube-api-access-tf7k4") pod "f67ffaea-1af3-455c-bc84-174dc116a709" (UID: "f67ffaea-1af3-455c-bc84-174dc116a709"). InnerVolumeSpecName "kube-api-access-tf7k4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:11:33.554229 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:33.554146    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f67ffaea-1af3-455c-bc84-174dc116a709-dshm\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:11:33.554229 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:33.554175    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f67ffaea-1af3-455c-bc84-174dc116a709-tls-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:11:33.554229 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:33.554187    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tf7k4\" (UniqueName: \"kubernetes.io/projected/f67ffaea-1af3-455c-bc84-174dc116a709-kube-api-access-tf7k4\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:11:33.975210 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:33.975173    2564 generic.go:356] "Generic (PLEG): container finished" podID="f67ffaea-1af3-455c-bc84-174dc116a709" containerID="11d6df5673d7251548ca2b30317569918d16eaefda36d976acbe8d9f8cecafb4" exitCode=0
Sep 18 20:11:33.975513 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:33.975223    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpcc6"
Sep 18 20:11:33.975513 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:33.975254    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpcc6" event={"ID":"f67ffaea-1af3-455c-bc84-174dc116a709","Type":"ContainerDied","Data":"11d6df5673d7251548ca2b30317569918d16eaefda36d976acbe8d9f8cecafb4"}
Sep 18 20:11:33.975513 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:33.975309    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpcc6" event={"ID":"f67ffaea-1af3-455c-bc84-174dc116a709","Type":"ContainerDied","Data":"8f9da872c18d78075d2f165df617cdca475310eb2e40d902a4084c56ef8ae955"}
Sep 18 20:11:33.975513 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:33.975331    2564 scope.go:117] "RemoveContainer" containerID="11d6df5673d7251548ca2b30317569918d16eaefda36d976acbe8d9f8cecafb4"
Sep 18 20:11:33.984529 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:33.984510    2564 scope.go:117] "RemoveContainer" containerID="11d6df5673d7251548ca2b30317569918d16eaefda36d976acbe8d9f8cecafb4"
Sep 18 20:11:33.984871 ip-10-0-128-161 kubenswrapper[2564]: E0918 20:11:33.984850    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11d6df5673d7251548ca2b30317569918d16eaefda36d976acbe8d9f8cecafb4\": container with ID starting with 11d6df5673d7251548ca2b30317569918d16eaefda36d976acbe8d9f8cecafb4 not found: ID does not exist" containerID="11d6df5673d7251548ca2b30317569918d16eaefda36d976acbe8d9f8cecafb4"
Sep 18 20:11:33.984950 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:33.984927    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11d6df5673d7251548ca2b30317569918d16eaefda36d976acbe8d9f8cecafb4"} err="failed to get container status \"11d6df5673d7251548ca2b30317569918d16eaefda36d976acbe8d9f8cecafb4\": rpc error: code = NotFound desc = could not find container \"11d6df5673d7251548ca2b30317569918d16eaefda36d976acbe8d9f8cecafb4\": container with ID starting with 11d6df5673d7251548ca2b30317569918d16eaefda36d976acbe8d9f8cecafb4 not found: ID does not exist"
Sep 18 20:11:34.001104 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:34.001065    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpcc6"]
Sep 18 20:11:34.001447 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:34.001425    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpcc6"]
Sep 18 20:11:36.000266 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:36.000227    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f67ffaea-1af3-455c-bc84-174dc116a709" path="/var/lib/kubelet/pods/f67ffaea-1af3-455c-bc84-174dc116a709/volumes"
Sep 18 20:11:38.873802 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:38.873766    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:38.975526 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:38.975484    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:39.074731 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:39.074695    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:40.108124 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:40.108090    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:43.300411 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:43.300372    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69bhj7h" podUID="34385379-07fe-4623-a216-f7b04275adad" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:11:43.947279 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:43.947245    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:45.037353 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:45.037307    2564 generic.go:356] "Generic (PLEG): container finished" podID="34385379-07fe-4623-a216-f7b04275adad" containerID="662533695d1af28970081cbac72b577f4c75f35da8cc9fe69a0b816353a06cb0" exitCode=0
Sep 18 20:11:45.037811 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:45.037376    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69bhj7h" event={"ID":"34385379-07fe-4623-a216-f7b04275adad","Type":"ContainerDied","Data":"662533695d1af28970081cbac72b577f4c75f35da8cc9fe69a0b816353a06cb0"}
Sep 18 20:11:45.106024 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:45.105934    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69bhj7h"
Sep 18 20:11:45.262827 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:45.262788    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34385379-07fe-4623-a216-f7b04275adad-tls-certs\") pod \"34385379-07fe-4623-a216-f7b04275adad\" (UID: \"34385379-07fe-4623-a216-f7b04275adad\") "
Sep 18 20:11:45.263072 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:45.262838    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fl44v\" (UniqueName: \"kubernetes.io/projected/34385379-07fe-4623-a216-f7b04275adad-kube-api-access-fl44v\") pod \"34385379-07fe-4623-a216-f7b04275adad\" (UID: \"34385379-07fe-4623-a216-f7b04275adad\") "
Sep 18 20:11:45.265249 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:45.265217    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34385379-07fe-4623-a216-f7b04275adad-kube-api-access-fl44v" (OuterVolumeSpecName: "kube-api-access-fl44v") pod "34385379-07fe-4623-a216-f7b04275adad" (UID: "34385379-07fe-4623-a216-f7b04275adad"). InnerVolumeSpecName "kube-api-access-fl44v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:11:45.265383 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:45.265256    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34385379-07fe-4623-a216-f7b04275adad-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "34385379-07fe-4623-a216-f7b04275adad" (UID: "34385379-07fe-4623-a216-f7b04275adad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:11:45.363862 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:45.363764    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34385379-07fe-4623-a216-f7b04275adad-tls-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:11:45.363862 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:45.363799    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fl44v\" (UniqueName: \"kubernetes.io/projected/34385379-07fe-4623-a216-f7b04275adad-kube-api-access-fl44v\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:11:46.044550 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:46.044524    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69bhj7h"
Sep 18 20:11:46.045055 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:46.044514    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69bhj7h" event={"ID":"34385379-07fe-4623-a216-f7b04275adad","Type":"ContainerDied","Data":"7e167f3136989c52d0c84daf4bf4b860d9d80ba521c6578ff77533de8478d89c"}
Sep 18 20:11:46.045055 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:46.044663    2564 scope.go:117] "RemoveContainer" containerID="662533695d1af28970081cbac72b577f4c75f35da8cc9fe69a0b816353a06cb0"
Sep 18 20:11:46.060956 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:46.060922    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69bhj7h"]
Sep 18 20:11:46.064556 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:46.064527    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69bhj7h"]
Sep 18 20:11:47.747339 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:47.747301    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:47.840646 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:47.840611    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:47.999433 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:47.999357    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34385379-07fe-4623-a216-f7b04275adad" path="/var/lib/kubelet/pods/34385379-07fe-4623-a216-f7b04275adad/volumes"
Sep 18 20:11:53.637294 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:53.637254    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:54.204183 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:54.204149    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-z272j"]
Sep 18 20:11:54.204592 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:54.204573    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f67ffaea-1af3-455c-bc84-174dc116a709" containerName="main"
Sep 18 20:11:54.204685 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:54.204595    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f67ffaea-1af3-455c-bc84-174dc116a709" containerName="main"
Sep 18 20:11:54.204685 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:54.204613    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34385379-07fe-4623-a216-f7b04275adad" containerName="main"
Sep 18 20:11:54.204685 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:54.204621    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="34385379-07fe-4623-a216-f7b04275adad" containerName="main"
Sep 18 20:11:54.204834 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:54.204742    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="34385379-07fe-4623-a216-f7b04275adad" containerName="main"
Sep 18 20:11:54.204834 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:54.204756    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f67ffaea-1af3-455c-bc84-174dc116a709" containerName="main"
Sep 18 20:11:54.209690 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:54.209665    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-z272j"
Sep 18 20:11:54.211828 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:54.211800    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"openshift-service-ca.crt\""
Sep 18 20:11:54.211828 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:54.211825    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"default-dockercfg-lcmpn\""
Sep 18 20:11:54.212441 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:54.212422    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"kube-root-ca.crt\""
Sep 18 20:11:54.212540 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:54.212524    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"decom-v1-kserve-self-signed-certs\""
Sep 18 20:11:54.219684 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:54.219659    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-z272j"]
Sep 18 20:11:54.236918 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:54.236878    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:54.242859 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:54.242824    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54725087-9db7-4029-88f7-dca1c1dc07a6-tmp-dir\") pod \"decom-v1-kserve-6cb797fb75-z272j\" (UID: \"54725087-9db7-4029-88f7-dca1c1dc07a6\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-z272j"
Sep 18 20:11:54.243028 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:54.242876    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54725087-9db7-4029-88f7-dca1c1dc07a6-home\") pod \"decom-v1-kserve-6cb797fb75-z272j\" (UID: \"54725087-9db7-4029-88f7-dca1c1dc07a6\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-z272j"
Sep 18 20:11:54.243028 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:54.242971    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54725087-9db7-4029-88f7-dca1c1dc07a6-dshm\") pod \"decom-v1-kserve-6cb797fb75-z272j\" (UID: \"54725087-9db7-4029-88f7-dca1c1dc07a6\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-z272j"
Sep 18 20:11:54.243122 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:54.243069    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54725087-9db7-4029-88f7-dca1c1dc07a6-tls-certs\") pod \"decom-v1-kserve-6cb797fb75-z272j\" (UID: \"54725087-9db7-4029-88f7-dca1c1dc07a6\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-z272j"
Sep 18 20:11:54.243122 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:54.243111    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54725087-9db7-4029-88f7-dca1c1dc07a6-model-cache\") pod \"decom-v1-kserve-6cb797fb75-z272j\" (UID: \"54725087-9db7-4029-88f7-dca1c1dc07a6\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-z272j"
Sep 18 20:11:54.243200 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:54.243182    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlx5x\" (UniqueName: \"kubernetes.io/projected/54725087-9db7-4029-88f7-dca1c1dc07a6-kube-api-access-vlx5x\") pod \"decom-v1-kserve-6cb797fb75-z272j\" (UID: \"54725087-9db7-4029-88f7-dca1c1dc07a6\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-z272j"
Sep 18 20:11:54.344166 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:54.344123    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54725087-9db7-4029-88f7-dca1c1dc07a6-home\") pod \"decom-v1-kserve-6cb797fb75-z272j\" (UID: \"54725087-9db7-4029-88f7-dca1c1dc07a6\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-z272j"
Sep 18 20:11:54.344360 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:54.344292    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54725087-9db7-4029-88f7-dca1c1dc07a6-dshm\") pod \"decom-v1-kserve-6cb797fb75-z272j\" (UID: \"54725087-9db7-4029-88f7-dca1c1dc07a6\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-z272j"
Sep 18 20:11:54.344360 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:54.344353    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54725087-9db7-4029-88f7-dca1c1dc07a6-tls-certs\") pod \"decom-v1-kserve-6cb797fb75-z272j\" (UID: \"54725087-9db7-4029-88f7-dca1c1dc07a6\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-z272j"
Sep 18 20:11:54.344451 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:54.344384    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54725087-9db7-4029-88f7-dca1c1dc07a6-model-cache\") pod \"decom-v1-kserve-6cb797fb75-z272j\" (UID: \"54725087-9db7-4029-88f7-dca1c1dc07a6\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-z272j"
Sep 18 20:11:54.344451 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:54.344435    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlx5x\" (UniqueName: \"kubernetes.io/projected/54725087-9db7-4029-88f7-dca1c1dc07a6-kube-api-access-vlx5x\") pod \"decom-v1-kserve-6cb797fb75-z272j\" (UID: \"54725087-9db7-4029-88f7-dca1c1dc07a6\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-z272j"
Sep 18 20:11:54.344554 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:54.344464    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54725087-9db7-4029-88f7-dca1c1dc07a6-tmp-dir\") pod \"decom-v1-kserve-6cb797fb75-z272j\" (UID: \"54725087-9db7-4029-88f7-dca1c1dc07a6\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-z272j"
Sep 18 20:11:54.344847 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:54.344814    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54725087-9db7-4029-88f7-dca1c1dc07a6-model-cache\") pod \"decom-v1-kserve-6cb797fb75-z272j\" (UID: \"54725087-9db7-4029-88f7-dca1c1dc07a6\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-z272j"
Sep 18 20:11:54.345011 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:54.344825    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54725087-9db7-4029-88f7-dca1c1dc07a6-tmp-dir\") pod \"decom-v1-kserve-6cb797fb75-z272j\" (UID: \"54725087-9db7-4029-88f7-dca1c1dc07a6\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-z272j"
Sep 18 20:11:54.345011 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:54.344938    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54725087-9db7-4029-88f7-dca1c1dc07a6-home\") pod \"decom-v1-kserve-6cb797fb75-z272j\" (UID: \"54725087-9db7-4029-88f7-dca1c1dc07a6\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-z272j"
Sep 18 20:11:54.346673 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:54.346653    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54725087-9db7-4029-88f7-dca1c1dc07a6-dshm\") pod \"decom-v1-kserve-6cb797fb75-z272j\" (UID: \"54725087-9db7-4029-88f7-dca1c1dc07a6\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-z272j"
Sep 18 20:11:54.347344 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:54.347323    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54725087-9db7-4029-88f7-dca1c1dc07a6-tls-certs\") pod \"decom-v1-kserve-6cb797fb75-z272j\" (UID: \"54725087-9db7-4029-88f7-dca1c1dc07a6\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-z272j"
Sep 18 20:11:54.352226 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:54.352197    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlx5x\" (UniqueName: \"kubernetes.io/projected/54725087-9db7-4029-88f7-dca1c1dc07a6-kube-api-access-vlx5x\") pod \"decom-v1-kserve-6cb797fb75-z272j\" (UID: \"54725087-9db7-4029-88f7-dca1c1dc07a6\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-z272j"
Sep 18 20:11:54.524440 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:54.524353    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-z272j"
Sep 18 20:11:54.807582 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:54.807489    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-z272j"]
Sep 18 20:11:54.811182 ip-10-0-128-161 kubenswrapper[2564]: W0918 20:11:54.811153    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54725087_9db7_4029_88f7_dca1c1dc07a6.slice/crio-5e9d6a59e16af5bf3c6f809a83a6a2d453b8c4c66156f962d12c1020239f1c0c WatchSource:0}: Error finding container 5e9d6a59e16af5bf3c6f809a83a6a2d453b8c4c66156f962d12c1020239f1c0c: Status 404 returned error can't find the container with id 5e9d6a59e16af5bf3c6f809a83a6a2d453b8c4c66156f962d12c1020239f1c0c
Sep 18 20:11:55.093318 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:55.093232    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-z272j" event={"ID":"54725087-9db7-4029-88f7-dca1c1dc07a6","Type":"ContainerStarted","Data":"5372cc7f63501c2ad0cfca6ea8a0056e041f1bdc061cd5436a3c04af50e74523"}
Sep 18 20:11:55.093318 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:55.093281    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-z272j" event={"ID":"54725087-9db7-4029-88f7-dca1c1dc07a6","Type":"ContainerStarted","Data":"5e9d6a59e16af5bf3c6f809a83a6a2d453b8c4c66156f962d12c1020239f1c0c"}
Sep 18 20:11:55.111434 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:55.111367    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-z272j" podStartSLOduration=1.111347141 podStartE2EDuration="1.111347141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:11:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:11:55.110065704 +0000 UTC m=+2031.620421815" watchObservedRunningTime="2026-09-18 20:11:55.111347141 +0000 UTC m=+2031.621703232"
Sep 18 20:11:55.524915 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:55.524877    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-z272j"
Sep 18 20:11:59.249493 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:59.249455    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:59.352977 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:11:59.352917    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:12:00.555229 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:12:00.555176    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:12:04.524976 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:12:04.524936    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-z272j"
Sep 18 20:12:04.538152 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:12:04.538117    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-z272j"
Sep 18 20:12:05.153025 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:12:05.152974    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-z272j"
Sep 18 20:12:13.541671 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:12:13.541633    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:12:13.831298 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:12:13.831210    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785k6s8t"]
Sep 18 20:12:13.836519 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:12:13.836493    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785k6s8t"
Sep 18 20:12:13.841645 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:12:13.841617    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 18 20:12:13.841645 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:12:13.841642    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 18 20:12:13.842804 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:12:13.842663    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-epp-sa-dockercfg-pg42v\""
Sep 18 20:12:13.842804 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:12:13.842688    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 18 20:12:13.842804 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:12:13.842692    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-7q8zm\""
Sep 18 20:12:13.851233 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:12:13.851201    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785k6s8t"]
Sep 18 20:12:13.867198 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:12:13.867164    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:12:13.920428 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:12:13.920393    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd7cm\" (UniqueName: \"kubernetes.io/projected/eec280a3-d5e2-448a-a7c6-eb948000ebd5-kube-api-access-jd7cm\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785k6s8t\" (UID: \"eec280a3-d5e2-448a-a7c6-eb948000ebd5\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785k6s8t"
Sep 18 20:12:13.920428 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:12:13.920433    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eec280a3-d5e2-448a-a7c6-eb948000ebd5-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785k6s8t\" (UID: \"eec280a3-d5e2-448a-a7c6-eb948000ebd5\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785k6s8t"
Sep 18 20:12:13.964761 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:12:13.964724    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:12:14.021208 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:12:14.021168    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd7cm\" (UniqueName: \"kubernetes.io/projected/eec280a3-d5e2-448a-a7c6-eb948000ebd5-kube-api-access-jd7cm\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785k6s8t\" (UID: \"eec280a3-d5e2-448a-a7c6-eb948000ebd5\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785k6s8t"
Sep 18 20:12:14.021208 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:12:14.021208    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eec280a3-d5e2-448a-a7c6-eb948000ebd5-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785k6s8t\" (UID: \"eec280a3-d5e2-448a-a7c6-eb948000ebd5\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785k6s8t"
Sep 18 20:12:14.023962 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:12:14.023940    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eec280a3-d5e2-448a-a7c6-eb948000ebd5-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785k6s8t\" (UID: \"eec280a3-d5e2-448a-a7c6-eb948000ebd5\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785k6s8t"
Sep 18 20:12:14.031001 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:12:14.030950    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd7cm\" (UniqueName: \"kubernetes.io/projected/eec280a3-d5e2-448a-a7c6-eb948000ebd5-kube-api-access-jd7cm\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785k6s8t\" (UID: \"eec280a3-d5e2-448a-a7c6-eb948000ebd5\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785k6s8t"
Sep 18 20:12:14.148388 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:12:14.148346    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785k6s8t"
Sep 18 20:12:14.423377 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:12:14.423347    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785k6s8t"]
Sep 18 20:12:14.425901 ip-10-0-128-161 kubenswrapper[2564]: W0918 20:12:14.425870    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeec280a3_d5e2_448a_a7c6_eb948000ebd5.slice/crio-5fa30a1a25473e5c36fe398a0490dad4ca10e23ac61dd96c25fb953a965e019a WatchSource:0}: Error finding container 5fa30a1a25473e5c36fe398a0490dad4ca10e23ac61dd96c25fb953a965e019a: Status 404 returned error can't find the container with id 5fa30a1a25473e5c36fe398a0490dad4ca10e23ac61dd96c25fb953a965e019a
Sep 18 20:12:15.192075 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:12:15.192025    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785k6s8t" event={"ID":"eec280a3-d5e2-448a-a7c6-eb948000ebd5","Type":"ContainerStarted","Data":"8e33728458868937e801c373eb4fb80d66a9de55b635bd70164695545fbb2849"}
Sep 18 20:12:15.192643 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:12:15.192094    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785k6s8t" event={"ID":"eec280a3-d5e2-448a-a7c6-eb948000ebd5","Type":"ContainerStarted","Data":"5fa30a1a25473e5c36fe398a0490dad4ca10e23ac61dd96c25fb953a965e019a"}
Sep 18 20:12:15.192643 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:12:15.192203    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785k6s8t"
Sep 18 20:12:15.213892 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:12:15.213842    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785k6s8t" podStartSLOduration=2.213824308 podStartE2EDuration="2.213824308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:12:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:12:15.212795819 +0000 UTC m=+2051.723151935" watchObservedRunningTime="2026-09-18 20:12:15.213824308 +0000 UTC m=+2051.724180399"
Sep 18 20:12:20.799509 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:12:20.799473    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:12:41.151693 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:12:41.151598    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:12:46.198423 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:12:46.198392    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785k6s8t"
Sep 18 20:12:56.422733 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:12:56.422691    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:12:56.438226 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:12:56.438185    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785k6s8t"]
Sep 18 20:12:56.496129 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:12:56.496092    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:12:57.146853 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:12:57.146811    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:12:58.517111 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:12:58.517074    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:04.059864 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:04.059830    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-58vl4_945a444e-9381-4e2f-b4c1-64be0ec88e6c/console-operator/2.log"
Sep 18 20:13:04.068471 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:04.068442    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-58vl4_945a444e-9381-4e2f-b4c1-64be0ec88e6c/console-operator/2.log"
Sep 18 20:13:06.199804 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:06.199758    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785k6s8t" podUID="eec280a3-d5e2-448a-a7c6-eb948000ebd5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:13:11.460779 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:11.460715    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785k6s8t" podUID="eec280a3-d5e2-448a-a7c6-eb948000ebd5" containerName="main" containerID="cri-o://8e33728458868937e801c373eb4fb80d66a9de55b635bd70164695545fbb2849" gracePeriod=45
Sep 18 20:13:14.026170 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:14.026128    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:14.130565 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:14.130525    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:14.231291 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:14.231251    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:14.360759 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:14.360653    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-z272j"]
Sep 18 20:13:14.393279 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:14.393240    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:16.199345 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:16.199307    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785k6s8t" podUID="eec280a3-d5e2-448a-a7c6-eb948000ebd5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:13:19.098566 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:19.098516    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:26.199357 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:26.199315    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785k6s8t" podUID="eec280a3-d5e2-448a-a7c6-eb948000ebd5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:13:26.199840 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:26.199463    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785k6s8t"
Sep 18 20:13:29.381262 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:29.381178    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-z272j" podUID="54725087-9db7-4029-88f7-dca1c1dc07a6" containerName="main" containerID="cri-o://5372cc7f63501c2ad0cfca6ea8a0056e041f1bdc061cd5436a3c04af50e74523" gracePeriod=45
Sep 18 20:13:29.570852 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:29.570816    2564 generic.go:356] "Generic (PLEG): container finished" podID="54725087-9db7-4029-88f7-dca1c1dc07a6" containerID="5372cc7f63501c2ad0cfca6ea8a0056e041f1bdc061cd5436a3c04af50e74523" exitCode=0
Sep 18 20:13:29.571025 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:29.570850    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-z272j" event={"ID":"54725087-9db7-4029-88f7-dca1c1dc07a6","Type":"ContainerDied","Data":"5372cc7f63501c2ad0cfca6ea8a0056e041f1bdc061cd5436a3c04af50e74523"}
Sep 18 20:13:29.639917 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:29.639850    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-z272j"
Sep 18 20:13:29.816456 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:29.816417    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54725087-9db7-4029-88f7-dca1c1dc07a6-tls-certs\") pod \"54725087-9db7-4029-88f7-dca1c1dc07a6\" (UID: \"54725087-9db7-4029-88f7-dca1c1dc07a6\") "
Sep 18 20:13:29.816635 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:29.816486    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54725087-9db7-4029-88f7-dca1c1dc07a6-dshm\") pod \"54725087-9db7-4029-88f7-dca1c1dc07a6\" (UID: \"54725087-9db7-4029-88f7-dca1c1dc07a6\") "
Sep 18 20:13:29.816635 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:29.816509    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vlx5x\" (UniqueName: \"kubernetes.io/projected/54725087-9db7-4029-88f7-dca1c1dc07a6-kube-api-access-vlx5x\") pod \"54725087-9db7-4029-88f7-dca1c1dc07a6\" (UID: \"54725087-9db7-4029-88f7-dca1c1dc07a6\") "
Sep 18 20:13:29.816635 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:29.816557    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54725087-9db7-4029-88f7-dca1c1dc07a6-model-cache\") pod \"54725087-9db7-4029-88f7-dca1c1dc07a6\" (UID: \"54725087-9db7-4029-88f7-dca1c1dc07a6\") "
Sep 18 20:13:29.816635 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:29.816600    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54725087-9db7-4029-88f7-dca1c1dc07a6-home\") pod \"54725087-9db7-4029-88f7-dca1c1dc07a6\" (UID: \"54725087-9db7-4029-88f7-dca1c1dc07a6\") "
Sep 18 20:13:29.816635 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:29.816633    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54725087-9db7-4029-88f7-dca1c1dc07a6-tmp-dir\") pod \"54725087-9db7-4029-88f7-dca1c1dc07a6\" (UID: \"54725087-9db7-4029-88f7-dca1c1dc07a6\") "
Sep 18 20:13:29.816907 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:29.816871    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54725087-9db7-4029-88f7-dca1c1dc07a6-home" (OuterVolumeSpecName: "home") pod "54725087-9db7-4029-88f7-dca1c1dc07a6" (UID: "54725087-9db7-4029-88f7-dca1c1dc07a6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:13:29.816907 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:29.816887    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54725087-9db7-4029-88f7-dca1c1dc07a6-model-cache" (OuterVolumeSpecName: "model-cache") pod "54725087-9db7-4029-88f7-dca1c1dc07a6" (UID: "54725087-9db7-4029-88f7-dca1c1dc07a6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:13:29.817045 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:29.816977    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54725087-9db7-4029-88f7-dca1c1dc07a6-model-cache\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:13:29.817045 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:29.817025    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54725087-9db7-4029-88f7-dca1c1dc07a6-home\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:13:29.817045 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:29.817022    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54725087-9db7-4029-88f7-dca1c1dc07a6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "54725087-9db7-4029-88f7-dca1c1dc07a6" (UID: "54725087-9db7-4029-88f7-dca1c1dc07a6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:13:29.818946 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:29.818924    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54725087-9db7-4029-88f7-dca1c1dc07a6-dshm" (OuterVolumeSpecName: "dshm") pod "54725087-9db7-4029-88f7-dca1c1dc07a6" (UID: "54725087-9db7-4029-88f7-dca1c1dc07a6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:13:29.819033 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:29.819008    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54725087-9db7-4029-88f7-dca1c1dc07a6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "54725087-9db7-4029-88f7-dca1c1dc07a6" (UID: "54725087-9db7-4029-88f7-dca1c1dc07a6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:13:29.819086 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:29.819064    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54725087-9db7-4029-88f7-dca1c1dc07a6-kube-api-access-vlx5x" (OuterVolumeSpecName: "kube-api-access-vlx5x") pod "54725087-9db7-4029-88f7-dca1c1dc07a6" (UID: "54725087-9db7-4029-88f7-dca1c1dc07a6"). InnerVolumeSpecName "kube-api-access-vlx5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:13:29.918276 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:29.918180    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54725087-9db7-4029-88f7-dca1c1dc07a6-dshm\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:13:29.918276 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:29.918210    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlx5x\" (UniqueName: \"kubernetes.io/projected/54725087-9db7-4029-88f7-dca1c1dc07a6-kube-api-access-vlx5x\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:13:29.918276 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:29.918240    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54725087-9db7-4029-88f7-dca1c1dc07a6-tmp-dir\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:13:29.918276 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:29.918250    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54725087-9db7-4029-88f7-dca1c1dc07a6-tls-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:13:30.253356 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:30.253256    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:30.578223 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:30.578140    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-z272j" event={"ID":"54725087-9db7-4029-88f7-dca1c1dc07a6","Type":"ContainerDied","Data":"5e9d6a59e16af5bf3c6f809a83a6a2d453b8c4c66156f962d12c1020239f1c0c"}
Sep 18 20:13:30.578223 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:30.578168    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-z272j"
Sep 18 20:13:30.578223 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:30.578203    2564 scope.go:117] "RemoveContainer" containerID="5372cc7f63501c2ad0cfca6ea8a0056e041f1bdc061cd5436a3c04af50e74523"
Sep 18 20:13:30.601664 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:30.601631    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-z272j"]
Sep 18 20:13:30.605661 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:30.605630    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-z272j"]
Sep 18 20:13:32.000889 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:32.000850    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54725087-9db7-4029-88f7-dca1c1dc07a6" path="/var/lib/kubelet/pods/54725087-9db7-4029-88f7-dca1c1dc07a6/volumes"
Sep 18 20:13:36.198965 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:36.198929    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785k6s8t" podUID="eec280a3-d5e2-448a-a7c6-eb948000ebd5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:13:36.595555 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:36.595448    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:41.634291 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:41.634251    2564 generic.go:356] "Generic (PLEG): container finished" podID="eec280a3-d5e2-448a-a7c6-eb948000ebd5" containerID="8e33728458868937e801c373eb4fb80d66a9de55b635bd70164695545fbb2849" exitCode=0
Sep 18 20:13:41.634628 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:41.634298    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785k6s8t" event={"ID":"eec280a3-d5e2-448a-a7c6-eb948000ebd5","Type":"ContainerDied","Data":"8e33728458868937e801c373eb4fb80d66a9de55b635bd70164695545fbb2849"}
Sep 18 20:13:41.718976 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:41.718911    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785k6s8t"
Sep 18 20:13:41.812352 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:41.812315    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jd7cm\" (UniqueName: \"kubernetes.io/projected/eec280a3-d5e2-448a-a7c6-eb948000ebd5-kube-api-access-jd7cm\") pod \"eec280a3-d5e2-448a-a7c6-eb948000ebd5\" (UID: \"eec280a3-d5e2-448a-a7c6-eb948000ebd5\") "
Sep 18 20:13:41.812548 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:41.812433    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eec280a3-d5e2-448a-a7c6-eb948000ebd5-tls-certs\") pod \"eec280a3-d5e2-448a-a7c6-eb948000ebd5\" (UID: \"eec280a3-d5e2-448a-a7c6-eb948000ebd5\") "
Sep 18 20:13:41.814659 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:41.814622    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eec280a3-d5e2-448a-a7c6-eb948000ebd5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "eec280a3-d5e2-448a-a7c6-eb948000ebd5" (UID: "eec280a3-d5e2-448a-a7c6-eb948000ebd5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:13:41.814790 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:41.814749    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eec280a3-d5e2-448a-a7c6-eb948000ebd5-kube-api-access-jd7cm" (OuterVolumeSpecName: "kube-api-access-jd7cm") pod "eec280a3-d5e2-448a-a7c6-eb948000ebd5" (UID: "eec280a3-d5e2-448a-a7c6-eb948000ebd5"). InnerVolumeSpecName "kube-api-access-jd7cm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:13:41.913117 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:41.913081    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jd7cm\" (UniqueName: \"kubernetes.io/projected/eec280a3-d5e2-448a-a7c6-eb948000ebd5-kube-api-access-jd7cm\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:13:41.913117 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:41.913112    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eec280a3-d5e2-448a-a7c6-eb948000ebd5-tls-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:13:42.640918 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:42.640890    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785k6s8t"
Sep 18 20:13:42.640918 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:42.640907    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785k6s8t" event={"ID":"eec280a3-d5e2-448a-a7c6-eb948000ebd5","Type":"ContainerDied","Data":"5fa30a1a25473e5c36fe398a0490dad4ca10e23ac61dd96c25fb953a965e019a"}
Sep 18 20:13:42.641438 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:42.640962    2564 scope.go:117] "RemoveContainer" containerID="8e33728458868937e801c373eb4fb80d66a9de55b635bd70164695545fbb2849"
Sep 18 20:13:42.667458 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:42.667421    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785k6s8t"]
Sep 18 20:13:42.672742 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:42.672714    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785k6s8t"]
Sep 18 20:13:44.001110 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:44.001078    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eec280a3-d5e2-448a-a7c6-eb948000ebd5" path="/var/lib/kubelet/pods/eec280a3-d5e2-448a-a7c6-eb948000ebd5/volumes"
Sep 18 20:13:44.567521 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:44.567479    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:44.756615 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:44.756579    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:51.027100 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:51.027058    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-vpkkb"]
Sep 18 20:13:51.027624 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:51.027478    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eec280a3-d5e2-448a-a7c6-eb948000ebd5" containerName="main"
Sep 18 20:13:51.027624 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:51.027492    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="eec280a3-d5e2-448a-a7c6-eb948000ebd5" containerName="main"
Sep 18 20:13:51.027624 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:51.027504    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54725087-9db7-4029-88f7-dca1c1dc07a6" containerName="main"
Sep 18 20:13:51.027624 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:51.027510    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="54725087-9db7-4029-88f7-dca1c1dc07a6" containerName="main"
Sep 18 20:13:51.027624 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:51.027571    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="eec280a3-d5e2-448a-a7c6-eb948000ebd5" containerName="main"
Sep 18 20:13:51.027624 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:51.027583    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="54725087-9db7-4029-88f7-dca1c1dc07a6" containerName="main"
Sep 18 20:13:51.031817 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:51.031794    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-vpkkb"
Sep 18 20:13:51.037327 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:51.037298    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"kube-root-ca.crt\""
Sep 18 20:13:51.037947 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:51.037918    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"default-dockercfg-9b2rl\""
Sep 18 20:13:51.038153 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:51.038133    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"leave-v2-kserve-self-signed-certs\""
Sep 18 20:13:51.038237 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:51.038185    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"openshift-service-ca.crt\""
Sep 18 20:13:51.047701 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:51.047665    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-vpkkb"]
Sep 18 20:13:51.076060 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:51.076019    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:51.090186 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:51.090148    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9dc70f9e-7d2f-471a-9c49-6f3d5e171b24-home\") pod \"leave-v2-kserve-556b4dddf9-vpkkb\" (UID: \"9dc70f9e-7d2f-471a-9c49-6f3d5e171b24\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-vpkkb"
Sep 18 20:13:51.090349 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:51.090261    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9dc70f9e-7d2f-471a-9c49-6f3d5e171b24-model-cache\") pod \"leave-v2-kserve-556b4dddf9-vpkkb\" (UID: \"9dc70f9e-7d2f-471a-9c49-6f3d5e171b24\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-vpkkb"
Sep 18 20:13:51.090349 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:51.090294    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7swsp\" (UniqueName: \"kubernetes.io/projected/9dc70f9e-7d2f-471a-9c49-6f3d5e171b24-kube-api-access-7swsp\") pod \"leave-v2-kserve-556b4dddf9-vpkkb\" (UID: \"9dc70f9e-7d2f-471a-9c49-6f3d5e171b24\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-vpkkb"
Sep 18 20:13:51.090451 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:51.090366    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dc70f9e-7d2f-471a-9c49-6f3d5e171b24-tmp-dir\") pod \"leave-v2-kserve-556b4dddf9-vpkkb\" (UID: \"9dc70f9e-7d2f-471a-9c49-6f3d5e171b24\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-vpkkb"
Sep 18 20:13:51.090451 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:51.090420    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dc70f9e-7d2f-471a-9c49-6f3d5e171b24-tls-certs\") pod \"leave-v2-kserve-556b4dddf9-vpkkb\" (UID: \"9dc70f9e-7d2f-471a-9c49-6f3d5e171b24\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-vpkkb"
Sep 18 20:13:51.090541 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:51.090483    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9dc70f9e-7d2f-471a-9c49-6f3d5e171b24-dshm\") pod \"leave-v2-kserve-556b4dddf9-vpkkb\" (UID: \"9dc70f9e-7d2f-471a-9c49-6f3d5e171b24\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-vpkkb"
Sep 18 20:13:51.191119 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:51.191069    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9dc70f9e-7d2f-471a-9c49-6f3d5e171b24-dshm\") pod \"leave-v2-kserve-556b4dddf9-vpkkb\" (UID: \"9dc70f9e-7d2f-471a-9c49-6f3d5e171b24\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-vpkkb"
Sep 18 20:13:51.191334 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:51.191133    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9dc70f9e-7d2f-471a-9c49-6f3d5e171b24-home\") pod \"leave-v2-kserve-556b4dddf9-vpkkb\" (UID: \"9dc70f9e-7d2f-471a-9c49-6f3d5e171b24\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-vpkkb"
Sep 18 20:13:51.191334 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:51.191186    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9dc70f9e-7d2f-471a-9c49-6f3d5e171b24-model-cache\") pod \"leave-v2-kserve-556b4dddf9-vpkkb\" (UID: \"9dc70f9e-7d2f-471a-9c49-6f3d5e171b24\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-vpkkb"
Sep 18 20:13:51.191334 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:51.191209    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7swsp\" (UniqueName: \"kubernetes.io/projected/9dc70f9e-7d2f-471a-9c49-6f3d5e171b24-kube-api-access-7swsp\") pod \"leave-v2-kserve-556b4dddf9-vpkkb\" (UID: \"9dc70f9e-7d2f-471a-9c49-6f3d5e171b24\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-vpkkb"
Sep 18 20:13:51.191334 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:51.191262    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dc70f9e-7d2f-471a-9c49-6f3d5e171b24-tmp-dir\") pod \"leave-v2-kserve-556b4dddf9-vpkkb\" (UID: \"9dc70f9e-7d2f-471a-9c49-6f3d5e171b24\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-vpkkb"
Sep 18 20:13:51.191334 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:51.191314    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dc70f9e-7d2f-471a-9c49-6f3d5e171b24-tls-certs\") pod \"leave-v2-kserve-556b4dddf9-vpkkb\" (UID: \"9dc70f9e-7d2f-471a-9c49-6f3d5e171b24\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-vpkkb"
Sep 18 20:13:51.191592 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:51.191570    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9dc70f9e-7d2f-471a-9c49-6f3d5e171b24-home\") pod \"leave-v2-kserve-556b4dddf9-vpkkb\" (UID: \"9dc70f9e-7d2f-471a-9c49-6f3d5e171b24\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-vpkkb"
Sep 18 20:13:51.191636 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:51.191620    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9dc70f9e-7d2f-471a-9c49-6f3d5e171b24-model-cache\") pod \"leave-v2-kserve-556b4dddf9-vpkkb\" (UID: \"9dc70f9e-7d2f-471a-9c49-6f3d5e171b24\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-vpkkb"
Sep 18 20:13:51.191728 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:51.191703    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dc70f9e-7d2f-471a-9c49-6f3d5e171b24-tmp-dir\") pod \"leave-v2-kserve-556b4dddf9-vpkkb\" (UID: \"9dc70f9e-7d2f-471a-9c49-6f3d5e171b24\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-vpkkb"
Sep 18 20:13:51.193464 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:51.193433    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9dc70f9e-7d2f-471a-9c49-6f3d5e171b24-dshm\") pod \"leave-v2-kserve-556b4dddf9-vpkkb\" (UID: \"9dc70f9e-7d2f-471a-9c49-6f3d5e171b24\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-vpkkb"
Sep 18 20:13:51.193793 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:51.193776    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dc70f9e-7d2f-471a-9c49-6f3d5e171b24-tls-certs\") pod \"leave-v2-kserve-556b4dddf9-vpkkb\" (UID: \"9dc70f9e-7d2f-471a-9c49-6f3d5e171b24\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-vpkkb"
Sep 18 20:13:51.201031 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:51.201003    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7swsp\" (UniqueName: \"kubernetes.io/projected/9dc70f9e-7d2f-471a-9c49-6f3d5e171b24-kube-api-access-7swsp\") pod \"leave-v2-kserve-556b4dddf9-vpkkb\" (UID: \"9dc70f9e-7d2f-471a-9c49-6f3d5e171b24\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-vpkkb"
Sep 18 20:13:51.345440 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:51.345318    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-vpkkb"
Sep 18 20:13:51.628219 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:51.628140    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-vpkkb"]
Sep 18 20:13:51.631529 ip-10-0-128-161 kubenswrapper[2564]: W0918 20:13:51.631502    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dc70f9e_7d2f_471a_9c49_6f3d5e171b24.slice/crio-911193b7f5d229bf863824ba84952fc29be42cf1ef773f83ad980faaae63b8ee WatchSource:0}: Error finding container 911193b7f5d229bf863824ba84952fc29be42cf1ef773f83ad980faaae63b8ee: Status 404 returned error can't find the container with id 911193b7f5d229bf863824ba84952fc29be42cf1ef773f83ad980faaae63b8ee
Sep 18 20:13:51.692387 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:51.692350    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-vpkkb" event={"ID":"9dc70f9e-7d2f-471a-9c49-6f3d5e171b24","Type":"ContainerStarted","Data":"911193b7f5d229bf863824ba84952fc29be42cf1ef773f83ad980faaae63b8ee"}
Sep 18 20:13:52.574628 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:52.574579    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:52.699324 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:52.699286    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-vpkkb" event={"ID":"9dc70f9e-7d2f-471a-9c49-6f3d5e171b24","Type":"ContainerStarted","Data":"67dbe8dd996917c2f417697b7e3ca0e32a6dc9284aeeafd746d03bc6f7b1bbf5"}
Sep 18 20:13:53.345646 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:53.345591    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-vpkkb"
Sep 18 20:13:57.430012 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:57.429884    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:59.456215 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:59.456161    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:59.642569 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:13:59.642530    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:01.345927 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:14:01.345874    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-vpkkb"
Sep 18 20:14:01.360542 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:14:01.360513    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-vpkkb"
Sep 18 20:14:01.413259 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:14:01.413204    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-vpkkb" podStartSLOduration=10.413185336 podStartE2EDuration="10.413185336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:13:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:13:52.744151561 +0000 UTC m=+2149.254507652" watchObservedRunningTime="2026-09-18 20:14:01.413185336 +0000 UTC m=+2157.923541428"
Sep 18 20:14:01.756637 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:14:01.756606    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-vpkkb"
Sep 18 20:14:04.255524 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:14:04.255456    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:04.522580 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:14:04.522475    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:19.519540 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:14:19.519501    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:19.988974 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:14:19.988936    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:19.991321 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:14:19.991297    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d9gs98"]
Sep 18 20:14:19.996162 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:14:19.996140    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d9gs98"
Sep 18 20:14:19.999655 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:14:19.999629    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 18 20:14:20.000846 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:14:20.000818    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 18 20:14:20.001042 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:14:20.001026    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-epp-sa-dockercfg-smzgk\""
Sep 18 20:14:20.001683 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:14:20.001663    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-4jbhf\""
Sep 18 20:14:20.001773 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:14:20.001683    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 18 20:14:20.021225 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:14:20.021194    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d9gs98"]
Sep 18 20:14:20.156458 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:14:20.156412    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e5c8cc1-29f2-424b-be3a-d753aabbc96c-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7d9gs98\" (UID: \"9e5c8cc1-29f2-424b-be3a-d753aabbc96c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d9gs98"
Sep 18 20:14:20.156667 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:14:20.156603    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9qtk\" (UniqueName: \"kubernetes.io/projected/9e5c8cc1-29f2-424b-be3a-d753aabbc96c-kube-api-access-h9qtk\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7d9gs98\" (UID: \"9e5c8cc1-29f2-424b-be3a-d753aabbc96c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d9gs98"
Sep 18 20:14:20.210762 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:14:20.210723    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:20.257885 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:14:20.257781    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e5c8cc1-29f2-424b-be3a-d753aabbc96c-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7d9gs98\" (UID: \"9e5c8cc1-29f2-424b-be3a-d753aabbc96c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d9gs98"
Sep 18 20:14:20.257885 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:14:20.257879    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9qtk\" (UniqueName: \"kubernetes.io/projected/9e5c8cc1-29f2-424b-be3a-d753aabbc96c-kube-api-access-h9qtk\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7d9gs98\" (UID: \"9e5c8cc1-29f2-424b-be3a-d753aabbc96c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d9gs98"
Sep 18 20:14:20.260407 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:14:20.260383    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e5c8cc1-29f2-424b-be3a-d753aabbc96c-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7d9gs98\" (UID: \"9e5c8cc1-29f2-424b-be3a-d753aabbc96c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d9gs98"
Sep 18 20:14:20.285039 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:14:20.284976    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9qtk\" (UniqueName: \"kubernetes.io/projected/9e5c8cc1-29f2-424b-be3a-d753aabbc96c-kube-api-access-h9qtk\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7d9gs98\" (UID: \"9e5c8cc1-29f2-424b-be3a-d753aabbc96c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d9gs98"
Sep 18 20:14:20.307750 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:14:20.307711    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d9gs98"
Sep 18 20:14:20.606014 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:14:20.605971    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d9gs98"]
Sep 18 20:14:20.607535 ip-10-0-128-161 kubenswrapper[2564]: W0918 20:14:20.607505    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e5c8cc1_29f2_424b_be3a_d753aabbc96c.slice/crio-7a975b9e0c7c49edf32a97713805dbc40678ce9a9dffe314171385ce0e946dfe WatchSource:0}: Error finding container 7a975b9e0c7c49edf32a97713805dbc40678ce9a9dffe314171385ce0e946dfe: Status 404 returned error can't find the container with id 7a975b9e0c7c49edf32a97713805dbc40678ce9a9dffe314171385ce0e946dfe
Sep 18 20:14:20.841265 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:14:20.841225    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d9gs98" event={"ID":"9e5c8cc1-29f2-424b-be3a-d753aabbc96c","Type":"ContainerStarted","Data":"af9f1484f4d2b3ba7ff047f87394a8449b3387a0d26ea99860b9da77e7605273"}
Sep 18 20:14:20.841452 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:14:20.841279    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d9gs98" event={"ID":"9e5c8cc1-29f2-424b-be3a-d753aabbc96c","Type":"ContainerStarted","Data":"7a975b9e0c7c49edf32a97713805dbc40678ce9a9dffe314171385ce0e946dfe"}
Sep 18 20:14:20.841452 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:14:20.841373    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d9gs98"
Sep 18 20:14:20.893761 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:14:20.893705    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d9gs98" podStartSLOduration=1.893688171 podStartE2EDuration="1.893688171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:14:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:14:20.892117731 +0000 UTC m=+2177.402473843" watchObservedRunningTime="2026-09-18 20:14:20.893688171 +0000 UTC m=+2177.404044261"
Sep 18 20:14:44.753773 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:14:44.753733    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:51.849761 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:14:51.849729    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d9gs98"
Sep 18 20:14:54.701745 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:14:54.701707    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:54.979553 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:14:54.979456    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:56.517604 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:14:56.517564    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-vpkkb"]
Sep 18 20:14:56.611807 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:14:56.611771    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:57.046828 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:14:57.046792    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:59.752167 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:14:59.752127    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:15:02.182454 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:02.182419    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:15:11.537163 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:11.537119    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-vpkkb" podUID="9dc70f9e-7d2f-471a-9c49-6f3d5e171b24" containerName="main" containerID="cri-o://67dbe8dd996917c2f417697b7e3ca0e32a6dc9284aeeafd746d03bc6f7b1bbf5" gracePeriod=45
Sep 18 20:15:11.794539 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:11.794467    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-vpkkb"
Sep 18 20:15:11.833995 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:11.833961    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dc70f9e-7d2f-471a-9c49-6f3d5e171b24-tmp-dir\") pod \"9dc70f9e-7d2f-471a-9c49-6f3d5e171b24\" (UID: \"9dc70f9e-7d2f-471a-9c49-6f3d5e171b24\") "
Sep 18 20:15:11.834182 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:11.834040    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9dc70f9e-7d2f-471a-9c49-6f3d5e171b24-home\") pod \"9dc70f9e-7d2f-471a-9c49-6f3d5e171b24\" (UID: \"9dc70f9e-7d2f-471a-9c49-6f3d5e171b24\") "
Sep 18 20:15:11.834182 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:11.834059    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7swsp\" (UniqueName: \"kubernetes.io/projected/9dc70f9e-7d2f-471a-9c49-6f3d5e171b24-kube-api-access-7swsp\") pod \"9dc70f9e-7d2f-471a-9c49-6f3d5e171b24\" (UID: \"9dc70f9e-7d2f-471a-9c49-6f3d5e171b24\") "
Sep 18 20:15:11.834182 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:11.834102    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9dc70f9e-7d2f-471a-9c49-6f3d5e171b24-model-cache\") pod \"9dc70f9e-7d2f-471a-9c49-6f3d5e171b24\" (UID: \"9dc70f9e-7d2f-471a-9c49-6f3d5e171b24\") "
Sep 18 20:15:11.834182 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:11.834129    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dc70f9e-7d2f-471a-9c49-6f3d5e171b24-tls-certs\") pod \"9dc70f9e-7d2f-471a-9c49-6f3d5e171b24\" (UID: \"9dc70f9e-7d2f-471a-9c49-6f3d5e171b24\") "
Sep 18 20:15:11.834182 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:11.834148    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9dc70f9e-7d2f-471a-9c49-6f3d5e171b24-dshm\") pod \"9dc70f9e-7d2f-471a-9c49-6f3d5e171b24\" (UID: \"9dc70f9e-7d2f-471a-9c49-6f3d5e171b24\") "
Sep 18 20:15:11.834442 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:11.834313    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dc70f9e-7d2f-471a-9c49-6f3d5e171b24-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9dc70f9e-7d2f-471a-9c49-6f3d5e171b24" (UID: "9dc70f9e-7d2f-471a-9c49-6f3d5e171b24"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:15:11.834442 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:11.834435    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dc70f9e-7d2f-471a-9c49-6f3d5e171b24-tmp-dir\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:15:11.834554 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:11.834478    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dc70f9e-7d2f-471a-9c49-6f3d5e171b24-model-cache" (OuterVolumeSpecName: "model-cache") pod "9dc70f9e-7d2f-471a-9c49-6f3d5e171b24" (UID: "9dc70f9e-7d2f-471a-9c49-6f3d5e171b24"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:15:11.834554 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:11.834507    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dc70f9e-7d2f-471a-9c49-6f3d5e171b24-home" (OuterVolumeSpecName: "home") pod "9dc70f9e-7d2f-471a-9c49-6f3d5e171b24" (UID: "9dc70f9e-7d2f-471a-9c49-6f3d5e171b24"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:15:11.836631 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:11.836591    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dc70f9e-7d2f-471a-9c49-6f3d5e171b24-dshm" (OuterVolumeSpecName: "dshm") pod "9dc70f9e-7d2f-471a-9c49-6f3d5e171b24" (UID: "9dc70f9e-7d2f-471a-9c49-6f3d5e171b24"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:15:11.836631 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:11.836605    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dc70f9e-7d2f-471a-9c49-6f3d5e171b24-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9dc70f9e-7d2f-471a-9c49-6f3d5e171b24" (UID: "9dc70f9e-7d2f-471a-9c49-6f3d5e171b24"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:15:11.836792 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:11.836708    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dc70f9e-7d2f-471a-9c49-6f3d5e171b24-kube-api-access-7swsp" (OuterVolumeSpecName: "kube-api-access-7swsp") pod "9dc70f9e-7d2f-471a-9c49-6f3d5e171b24" (UID: "9dc70f9e-7d2f-471a-9c49-6f3d5e171b24"). InnerVolumeSpecName "kube-api-access-7swsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:15:11.935275 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:11.935238    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dc70f9e-7d2f-471a-9c49-6f3d5e171b24-tls-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:15:11.935275 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:11.935267    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9dc70f9e-7d2f-471a-9c49-6f3d5e171b24-dshm\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:15:11.935275 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:11.935276    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9dc70f9e-7d2f-471a-9c49-6f3d5e171b24-home\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:15:11.935529 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:11.935284    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7swsp\" (UniqueName: \"kubernetes.io/projected/9dc70f9e-7d2f-471a-9c49-6f3d5e171b24-kube-api-access-7swsp\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:15:11.935529 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:11.935296    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9dc70f9e-7d2f-471a-9c49-6f3d5e171b24-model-cache\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:15:12.107413 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:12.107325    2564 generic.go:356] "Generic (PLEG): container finished" podID="9dc70f9e-7d2f-471a-9c49-6f3d5e171b24" containerID="67dbe8dd996917c2f417697b7e3ca0e32a6dc9284aeeafd746d03bc6f7b1bbf5" exitCode=0
Sep 18 20:15:12.107413 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:12.107374    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-vpkkb"
Sep 18 20:15:12.107413 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:12.107398    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-vpkkb" event={"ID":"9dc70f9e-7d2f-471a-9c49-6f3d5e171b24","Type":"ContainerDied","Data":"67dbe8dd996917c2f417697b7e3ca0e32a6dc9284aeeafd746d03bc6f7b1bbf5"}
Sep 18 20:15:12.107668 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:12.107440    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-vpkkb" event={"ID":"9dc70f9e-7d2f-471a-9c49-6f3d5e171b24","Type":"ContainerDied","Data":"911193b7f5d229bf863824ba84952fc29be42cf1ef773f83ad980faaae63b8ee"}
Sep 18 20:15:12.107668 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:12.107462    2564 scope.go:117] "RemoveContainer" containerID="67dbe8dd996917c2f417697b7e3ca0e32a6dc9284aeeafd746d03bc6f7b1bbf5"
Sep 18 20:15:12.116118 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:12.116092    2564 scope.go:117] "RemoveContainer" containerID="67dbe8dd996917c2f417697b7e3ca0e32a6dc9284aeeafd746d03bc6f7b1bbf5"
Sep 18 20:15:12.116420 ip-10-0-128-161 kubenswrapper[2564]: E0918 20:15:12.116399    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67dbe8dd996917c2f417697b7e3ca0e32a6dc9284aeeafd746d03bc6f7b1bbf5\": container with ID starting with 67dbe8dd996917c2f417697b7e3ca0e32a6dc9284aeeafd746d03bc6f7b1bbf5 not found: ID does not exist" containerID="67dbe8dd996917c2f417697b7e3ca0e32a6dc9284aeeafd746d03bc6f7b1bbf5"
Sep 18 20:15:12.116482 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:12.116428    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67dbe8dd996917c2f417697b7e3ca0e32a6dc9284aeeafd746d03bc6f7b1bbf5"} err="failed to get container status \"67dbe8dd996917c2f417697b7e3ca0e32a6dc9284aeeafd746d03bc6f7b1bbf5\": rpc error: code = NotFound desc = could not find container \"67dbe8dd996917c2f417697b7e3ca0e32a6dc9284aeeafd746d03bc6f7b1bbf5\": container with ID starting with 67dbe8dd996917c2f417697b7e3ca0e32a6dc9284aeeafd746d03bc6f7b1bbf5 not found: ID does not exist"
Sep 18 20:15:12.139360 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:12.139324    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-vpkkb"]
Sep 18 20:15:12.145478 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:12.145445    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-vpkkb"]
Sep 18 20:15:12.746023 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:12.745941    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-vpkkb" podUID="9dc70f9e-7d2f-471a-9c49-6f3d5e171b24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": context deadline exceeded"
Sep 18 20:15:13.623810 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:13.623777    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:15:13.723364 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:13.723326    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:15:14.001698 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:14.001663    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9dc70f9e-7d2f-471a-9c49-6f3d5e171b24" path="/var/lib/kubelet/pods/9dc70f9e-7d2f-471a-9c49-6f3d5e171b24/volumes"
Sep 18 20:15:36.089267 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:36.089170    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:15:40.509407 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:40.509370    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:15:42.929668 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:42.929633    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7gqh5"]
Sep 18 20:15:42.930075 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:42.930060    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9dc70f9e-7d2f-471a-9c49-6f3d5e171b24" containerName="main"
Sep 18 20:15:42.930124 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:42.930076    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9dc70f9e-7d2f-471a-9c49-6f3d5e171b24" containerName="main"
Sep 18 20:15:42.930160 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:42.930135    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9dc70f9e-7d2f-471a-9c49-6f3d5e171b24" containerName="main"
Sep 18 20:15:42.933207 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:42.933187    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7gqh5"
Sep 18 20:15:42.944742 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:42.944719    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"tri-v3-kserve-self-signed-certs\""
Sep 18 20:15:42.945365 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:42.945350    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"kube-root-ca.crt\""
Sep 18 20:15:42.945427 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:42.945360    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"default-dockercfg-xtbjf\""
Sep 18 20:15:42.945427 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:42.945412    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"openshift-service-ca.crt\""
Sep 18 20:15:42.956535 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:42.956496    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7gqh5"]
Sep 18 20:15:42.989253 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:42.989212    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:15:43.016694 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:43.016653    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4358682f-db65-4e80-b47a-741cfd02d5a4-dshm\") pod \"tri-v3-kserve-798c88c66c-7gqh5\" (UID: \"4358682f-db65-4e80-b47a-741cfd02d5a4\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7gqh5"
Sep 18 20:15:43.016694 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:43.016693    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4358682f-db65-4e80-b47a-741cfd02d5a4-home\") pod \"tri-v3-kserve-798c88c66c-7gqh5\" (UID: \"4358682f-db65-4e80-b47a-741cfd02d5a4\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7gqh5"
Sep 18 20:15:43.016921 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:43.016814    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4358682f-db65-4e80-b47a-741cfd02d5a4-tmp-dir\") pod \"tri-v3-kserve-798c88c66c-7gqh5\" (UID: \"4358682f-db65-4e80-b47a-741cfd02d5a4\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7gqh5"
Sep 18 20:15:43.016921 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:43.016854    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4358682f-db65-4e80-b47a-741cfd02d5a4-tls-certs\") pod \"tri-v3-kserve-798c88c66c-7gqh5\" (UID: \"4358682f-db65-4e80-b47a-741cfd02d5a4\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7gqh5"
Sep 18 20:15:43.017045 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:43.016913    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4358682f-db65-4e80-b47a-741cfd02d5a4-model-cache\") pod \"tri-v3-kserve-798c88c66c-7gqh5\" (UID: \"4358682f-db65-4e80-b47a-741cfd02d5a4\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7gqh5"
Sep 18 20:15:43.017045 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:43.016951    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4l65\" (UniqueName: \"kubernetes.io/projected/4358682f-db65-4e80-b47a-741cfd02d5a4-kube-api-access-j4l65\") pod \"tri-v3-kserve-798c88c66c-7gqh5\" (UID: \"4358682f-db65-4e80-b47a-741cfd02d5a4\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7gqh5"
Sep 18 20:15:43.118100 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:43.118062    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4358682f-db65-4e80-b47a-741cfd02d5a4-tmp-dir\") pod \"tri-v3-kserve-798c88c66c-7gqh5\" (UID: \"4358682f-db65-4e80-b47a-741cfd02d5a4\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7gqh5"
Sep 18 20:15:43.118316 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:43.118118    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4358682f-db65-4e80-b47a-741cfd02d5a4-tls-certs\") pod \"tri-v3-kserve-798c88c66c-7gqh5\" (UID: \"4358682f-db65-4e80-b47a-741cfd02d5a4\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7gqh5"
Sep 18 20:15:43.118316 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:43.118159    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4358682f-db65-4e80-b47a-741cfd02d5a4-model-cache\") pod \"tri-v3-kserve-798c88c66c-7gqh5\" (UID: \"4358682f-db65-4e80-b47a-741cfd02d5a4\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7gqh5"
Sep 18 20:15:43.118316 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:43.118199    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4l65\" (UniqueName: \"kubernetes.io/projected/4358682f-db65-4e80-b47a-741cfd02d5a4-kube-api-access-j4l65\") pod \"tri-v3-kserve-798c88c66c-7gqh5\" (UID: \"4358682f-db65-4e80-b47a-741cfd02d5a4\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7gqh5"
Sep 18 20:15:43.118316 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:43.118237    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4358682f-db65-4e80-b47a-741cfd02d5a4-dshm\") pod \"tri-v3-kserve-798c88c66c-7gqh5\" (UID: \"4358682f-db65-4e80-b47a-741cfd02d5a4\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7gqh5"
Sep 18 20:15:43.118316 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:43.118264    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4358682f-db65-4e80-b47a-741cfd02d5a4-home\") pod \"tri-v3-kserve-798c88c66c-7gqh5\" (UID: \"4358682f-db65-4e80-b47a-741cfd02d5a4\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7gqh5"
Sep 18 20:15:43.118599 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:43.118568    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4358682f-db65-4e80-b47a-741cfd02d5a4-tmp-dir\") pod \"tri-v3-kserve-798c88c66c-7gqh5\" (UID: \"4358682f-db65-4e80-b47a-741cfd02d5a4\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7gqh5"
Sep 18 20:15:43.118680 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:43.118657    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4358682f-db65-4e80-b47a-741cfd02d5a4-model-cache\") pod \"tri-v3-kserve-798c88c66c-7gqh5\" (UID: \"4358682f-db65-4e80-b47a-741cfd02d5a4\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7gqh5"
Sep 18 20:15:43.118740 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:43.118695    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4358682f-db65-4e80-b47a-741cfd02d5a4-home\") pod \"tri-v3-kserve-798c88c66c-7gqh5\" (UID: \"4358682f-db65-4e80-b47a-741cfd02d5a4\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7gqh5"
Sep 18 20:15:43.121640 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:43.121588    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4358682f-db65-4e80-b47a-741cfd02d5a4-tls-certs\") pod \"tri-v3-kserve-798c88c66c-7gqh5\" (UID: \"4358682f-db65-4e80-b47a-741cfd02d5a4\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7gqh5"
Sep 18 20:15:43.122145 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:43.122121    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4358682f-db65-4e80-b47a-741cfd02d5a4-dshm\") pod \"tri-v3-kserve-798c88c66c-7gqh5\" (UID: \"4358682f-db65-4e80-b47a-741cfd02d5a4\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7gqh5"
Sep 18 20:15:43.132634 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:43.132597    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4l65\" (UniqueName: \"kubernetes.io/projected/4358682f-db65-4e80-b47a-741cfd02d5a4-kube-api-access-j4l65\") pod \"tri-v3-kserve-798c88c66c-7gqh5\" (UID: \"4358682f-db65-4e80-b47a-741cfd02d5a4\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7gqh5"
Sep 18 20:15:43.244265 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:43.244165    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7gqh5"
Sep 18 20:15:43.529401 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:43.529372    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7gqh5"]
Sep 18 20:15:43.532153 ip-10-0-128-161 kubenswrapper[2564]: W0918 20:15:43.532124    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4358682f_db65_4e80_b47a_741cfd02d5a4.slice/crio-bf95fcca91dc72625e5eaca9e3ecc0bca516fb083ffa00ce1d455287caa26a63 WatchSource:0}: Error finding container bf95fcca91dc72625e5eaca9e3ecc0bca516fb083ffa00ce1d455287caa26a63: Status 404 returned error can't find the container with id bf95fcca91dc72625e5eaca9e3ecc0bca516fb083ffa00ce1d455287caa26a63
Sep 18 20:15:44.268537 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:44.268491    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7gqh5" event={"ID":"4358682f-db65-4e80-b47a-741cfd02d5a4","Type":"ContainerStarted","Data":"c8cb298fca6f310dafe862229ce7c941cf6d8e0de7a4010300b3799acb1ee641"}
Sep 18 20:15:44.270963 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:44.268558    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7gqh5" event={"ID":"4358682f-db65-4e80-b47a-741cfd02d5a4","Type":"ContainerStarted","Data":"bf95fcca91dc72625e5eaca9e3ecc0bca516fb083ffa00ce1d455287caa26a63"}
Sep 18 20:15:45.244793 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:45.244753    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7gqh5"
Sep 18 20:15:46.689409 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:46.689363    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:15:46.793641 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:46.793600    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:15:53.244762 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:53.244720    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7gqh5"
Sep 18 20:15:53.258770 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:53.258740    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7gqh5"
Sep 18 20:15:53.278842 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:53.278789    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7gqh5" podStartSLOduration=11.278771138 podStartE2EDuration="11.278771138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:15:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:15:44.288885759 +0000 UTC m=+2260.799241897" watchObservedRunningTime="2026-09-18 20:15:53.278771138 +0000 UTC m=+2269.789127229"
Sep 18 20:15:53.344043 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:15:53.343976    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7gqh5"
Sep 18 20:16:18.818659 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:16:18.818616    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:16:18.836359 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:16:18.836325    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d9gs98"]
Sep 18 20:16:18.899754 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:16:18.899701    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:16:21.850086 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:16:21.850045    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d9gs98" podUID="9e5c8cc1-29f2-424b-be3a-d753aabbc96c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:16:31.849551 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:16:31.849511    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d9gs98" podUID="9e5c8cc1-29f2-424b-be3a-d753aabbc96c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:16:33.858144 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:16:33.858093    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d9gs98" podUID="9e5c8cc1-29f2-424b-be3a-d753aabbc96c" containerName="main" containerID="cri-o://af9f1484f4d2b3ba7ff047f87394a8449b3387a0d26ea99860b9da77e7605273" gracePeriod=45
Sep 18 20:16:41.849817 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:16:41.849777    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d9gs98" podUID="9e5c8cc1-29f2-424b-be3a-d753aabbc96c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:16:41.850331 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:16:41.849902    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d9gs98"
Sep 18 20:16:51.849329 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:16:51.849290    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d9gs98" podUID="9e5c8cc1-29f2-424b-be3a-d753aabbc96c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:17:01.850150 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:01.850057    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d9gs98" podUID="9e5c8cc1-29f2-424b-be3a-d753aabbc96c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:17:04.111050 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:04.111024    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d9gs98"
Sep 18 20:17:04.149910 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:04.149870    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e5c8cc1-29f2-424b-be3a-d753aabbc96c-tls-certs\") pod \"9e5c8cc1-29f2-424b-be3a-d753aabbc96c\" (UID: \"9e5c8cc1-29f2-424b-be3a-d753aabbc96c\") "
Sep 18 20:17:04.150097 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:04.150074    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h9qtk\" (UniqueName: \"kubernetes.io/projected/9e5c8cc1-29f2-424b-be3a-d753aabbc96c-kube-api-access-h9qtk\") pod \"9e5c8cc1-29f2-424b-be3a-d753aabbc96c\" (UID: \"9e5c8cc1-29f2-424b-be3a-d753aabbc96c\") "
Sep 18 20:17:04.152370 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:04.152332    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e5c8cc1-29f2-424b-be3a-d753aabbc96c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9e5c8cc1-29f2-424b-be3a-d753aabbc96c" (UID: "9e5c8cc1-29f2-424b-be3a-d753aabbc96c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:17:04.152505 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:04.152388    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e5c8cc1-29f2-424b-be3a-d753aabbc96c-kube-api-access-h9qtk" (OuterVolumeSpecName: "kube-api-access-h9qtk") pod "9e5c8cc1-29f2-424b-be3a-d753aabbc96c" (UID: "9e5c8cc1-29f2-424b-be3a-d753aabbc96c"). InnerVolumeSpecName "kube-api-access-h9qtk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:17:04.251370 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:04.251328    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9qtk\" (UniqueName: \"kubernetes.io/projected/9e5c8cc1-29f2-424b-be3a-d753aabbc96c-kube-api-access-h9qtk\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:17:04.251370 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:04.251365    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e5c8cc1-29f2-424b-be3a-d753aabbc96c-tls-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:17:04.682957 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:04.682924    2564 generic.go:356] "Generic (PLEG): container finished" podID="9e5c8cc1-29f2-424b-be3a-d753aabbc96c" containerID="af9f1484f4d2b3ba7ff047f87394a8449b3387a0d26ea99860b9da77e7605273" exitCode=0
Sep 18 20:17:04.683181 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:04.682968    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d9gs98"
Sep 18 20:17:04.683181 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:04.683017    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d9gs98" event={"ID":"9e5c8cc1-29f2-424b-be3a-d753aabbc96c","Type":"ContainerDied","Data":"af9f1484f4d2b3ba7ff047f87394a8449b3387a0d26ea99860b9da77e7605273"}
Sep 18 20:17:04.683181 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:04.683076    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d9gs98" event={"ID":"9e5c8cc1-29f2-424b-be3a-d753aabbc96c","Type":"ContainerDied","Data":"7a975b9e0c7c49edf32a97713805dbc40678ce9a9dffe314171385ce0e946dfe"}
Sep 18 20:17:04.683181 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:04.683100    2564 scope.go:117] "RemoveContainer" containerID="af9f1484f4d2b3ba7ff047f87394a8449b3387a0d26ea99860b9da77e7605273"
Sep 18 20:17:04.692264 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:04.692244    2564 scope.go:117] "RemoveContainer" containerID="af9f1484f4d2b3ba7ff047f87394a8449b3387a0d26ea99860b9da77e7605273"
Sep 18 20:17:04.692555 ip-10-0-128-161 kubenswrapper[2564]: E0918 20:17:04.692538    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af9f1484f4d2b3ba7ff047f87394a8449b3387a0d26ea99860b9da77e7605273\": container with ID starting with af9f1484f4d2b3ba7ff047f87394a8449b3387a0d26ea99860b9da77e7605273 not found: ID does not exist" containerID="af9f1484f4d2b3ba7ff047f87394a8449b3387a0d26ea99860b9da77e7605273"
Sep 18 20:17:04.692600 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:04.692563    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af9f1484f4d2b3ba7ff047f87394a8449b3387a0d26ea99860b9da77e7605273"} err="failed to get container status \"af9f1484f4d2b3ba7ff047f87394a8449b3387a0d26ea99860b9da77e7605273\": rpc error: code = NotFound desc = could not find container \"af9f1484f4d2b3ba7ff047f87394a8449b3387a0d26ea99860b9da77e7605273\": container with ID starting with af9f1484f4d2b3ba7ff047f87394a8449b3387a0d26ea99860b9da77e7605273 not found: ID does not exist"
Sep 18 20:17:04.704120 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:04.704088    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d9gs98"]
Sep 18 20:17:04.708052 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:04.708010    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d9gs98"]
Sep 18 20:17:06.006476 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:06.006435    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e5c8cc1-29f2-424b-be3a-d753aabbc96c" path="/var/lib/kubelet/pods/9e5c8cc1-29f2-424b-be3a-d753aabbc96c/volumes"
Sep 18 20:17:07.058393 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:07.058353    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:07.149718 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:07.149683    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:11.549891 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:11.549855    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:11.736612 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:11.736570    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezlnbn"]
Sep 18 20:17:11.737176 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:11.737153    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e5c8cc1-29f2-424b-be3a-d753aabbc96c" containerName="main"
Sep 18 20:17:11.737176 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:11.737174    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e5c8cc1-29f2-424b-be3a-d753aabbc96c" containerName="main"
Sep 18 20:17:11.737374 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:11.737265    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e5c8cc1-29f2-424b-be3a-d753aabbc96c" containerName="main"
Sep 18 20:17:11.740159 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:11.740136    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezlnbn"
Sep 18 20:17:11.743466 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:11.743182    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Sep 18 20:17:11.743466 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:11.743206    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Sep 18 20:17:11.743466 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:11.743218    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Sep 18 20:17:11.743466 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:11.743231    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-lv8nw\""
Sep 18 20:17:11.743466 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:11.743390    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisvc-router-managed-test-llm-e45d1f79-epp-sa-dockercfg-w6kbt\""
Sep 18 20:17:11.750263 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:11.750237    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezlnbn"]
Sep 18 20:17:11.768144 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:11.768104    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:11.819555 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:11.819450    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg6n6\" (UniqueName: \"kubernetes.io/projected/9ccdba2d-cb81-4f9d-9378-c35b93e1474d-kube-api-access-vg6n6\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezlnbn\" (UID: \"9ccdba2d-cb81-4f9d-9378-c35b93e1474d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezlnbn"
Sep 18 20:17:11.819555 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:11.819501    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ccdba2d-cb81-4f9d-9378-c35b93e1474d-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezlnbn\" (UID: \"9ccdba2d-cb81-4f9d-9378-c35b93e1474d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezlnbn"
Sep 18 20:17:11.919943 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:11.919899    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg6n6\" (UniqueName: \"kubernetes.io/projected/9ccdba2d-cb81-4f9d-9378-c35b93e1474d-kube-api-access-vg6n6\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezlnbn\" (UID: \"9ccdba2d-cb81-4f9d-9378-c35b93e1474d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezlnbn"
Sep 18 20:17:11.919943 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:11.919947    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ccdba2d-cb81-4f9d-9378-c35b93e1474d-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezlnbn\" (UID: \"9ccdba2d-cb81-4f9d-9378-c35b93e1474d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezlnbn"
Sep 18 20:17:11.922596 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:11.922555    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ccdba2d-cb81-4f9d-9378-c35b93e1474d-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezlnbn\" (UID: \"9ccdba2d-cb81-4f9d-9378-c35b93e1474d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezlnbn"
Sep 18 20:17:11.928359 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:11.928333    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg6n6\" (UniqueName: \"kubernetes.io/projected/9ccdba2d-cb81-4f9d-9378-c35b93e1474d-kube-api-access-vg6n6\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezlnbn\" (UID: \"9ccdba2d-cb81-4f9d-9378-c35b93e1474d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezlnbn"
Sep 18 20:17:12.052612 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:12.052571    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezlnbn"
Sep 18 20:17:12.330718 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:12.329596    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezlnbn"]
Sep 18 20:17:12.335771 ip-10-0-128-161 kubenswrapper[2564]: W0918 20:17:12.335741    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ccdba2d_cb81_4f9d_9378_c35b93e1474d.slice/crio-05c5fc7ff4cc5ac4c40cacca83ef5d711b54d4e580bc0c3288511144a383d4dd WatchSource:0}: Error finding container 05c5fc7ff4cc5ac4c40cacca83ef5d711b54d4e580bc0c3288511144a383d4dd: Status 404 returned error can't find the container with id 05c5fc7ff4cc5ac4c40cacca83ef5d711b54d4e580bc0c3288511144a383d4dd
Sep 18 20:17:12.724533 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:12.724495    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezlnbn" event={"ID":"9ccdba2d-cb81-4f9d-9378-c35b93e1474d","Type":"ContainerStarted","Data":"b56f8f42b658ff23a21eac1f1ea8ae361b5cda69c0f5d4fa4b3be190cfc6d6bb"}
Sep 18 20:17:12.724955 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:12.724549    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezlnbn" event={"ID":"9ccdba2d-cb81-4f9d-9378-c35b93e1474d","Type":"ContainerStarted","Data":"05c5fc7ff4cc5ac4c40cacca83ef5d711b54d4e580bc0c3288511144a383d4dd"}
Sep 18 20:17:12.724955 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:12.724593    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezlnbn"
Sep 18 20:17:12.742083 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:12.742026    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezlnbn" podStartSLOduration=1.742007746 podStartE2EDuration="1.742007746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:17:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:17:12.739442667 +0000 UTC m=+2349.249798758" watchObservedRunningTime="2026-09-18 20:17:12.742007746 +0000 UTC m=+2349.252363841"
Sep 18 20:17:13.794711 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:13.794672    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:14.445195 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:14.445159    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:14.543670 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:14.543633    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:14.644634 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:14.644596    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:15.436175 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:15.436140    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:19.425745 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:19.425705    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7gqh5"]
Sep 18 20:17:19.516244 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:19.516189    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:19.615181 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:19.615144    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:19.717119 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:19.717032    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:26.705012 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:26.704952    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:28.322249 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:28.322204    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:29.108929 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:29.108894    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:34.447598 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:34.447520    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7gqh5" podUID="4358682f-db65-4e80-b47a-741cfd02d5a4" containerName="main" containerID="cri-o://c8cb298fca6f310dafe862229ce7c941cf6d8e0de7a4010300b3799acb1ee641" gracePeriod=45
Sep 18 20:17:34.711254 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:34.711176    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7gqh5"
Sep 18 20:17:34.834604 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:34.834574    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4358682f-db65-4e80-b47a-741cfd02d5a4-tls-certs\") pod \"4358682f-db65-4e80-b47a-741cfd02d5a4\" (UID: \"4358682f-db65-4e80-b47a-741cfd02d5a4\") "
Sep 18 20:17:34.834824 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:34.834645    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4358682f-db65-4e80-b47a-741cfd02d5a4-tmp-dir\") pod \"4358682f-db65-4e80-b47a-741cfd02d5a4\" (UID: \"4358682f-db65-4e80-b47a-741cfd02d5a4\") "
Sep 18 20:17:34.834824 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:34.834750    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4358682f-db65-4e80-b47a-741cfd02d5a4-dshm\") pod \"4358682f-db65-4e80-b47a-741cfd02d5a4\" (UID: \"4358682f-db65-4e80-b47a-741cfd02d5a4\") "
Sep 18 20:17:34.834824 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:34.834807    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4358682f-db65-4e80-b47a-741cfd02d5a4-home\") pod \"4358682f-db65-4e80-b47a-741cfd02d5a4\" (UID: \"4358682f-db65-4e80-b47a-741cfd02d5a4\") "
Sep 18 20:17:34.835033 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:34.834836    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4358682f-db65-4e80-b47a-741cfd02d5a4-model-cache\") pod \"4358682f-db65-4e80-b47a-741cfd02d5a4\" (UID: \"4358682f-db65-4e80-b47a-741cfd02d5a4\") "
Sep 18 20:17:34.835033 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:34.834874    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j4l65\" (UniqueName: \"kubernetes.io/projected/4358682f-db65-4e80-b47a-741cfd02d5a4-kube-api-access-j4l65\") pod \"4358682f-db65-4e80-b47a-741cfd02d5a4\" (UID: \"4358682f-db65-4e80-b47a-741cfd02d5a4\") "
Sep 18 20:17:34.835033 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:34.834942    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4358682f-db65-4e80-b47a-741cfd02d5a4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4358682f-db65-4e80-b47a-741cfd02d5a4" (UID: "4358682f-db65-4e80-b47a-741cfd02d5a4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:17:34.835207 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:34.835131    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4358682f-db65-4e80-b47a-741cfd02d5a4-home" (OuterVolumeSpecName: "home") pod "4358682f-db65-4e80-b47a-741cfd02d5a4" (UID: "4358682f-db65-4e80-b47a-741cfd02d5a4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:17:34.835207 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:34.835191    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4358682f-db65-4e80-b47a-741cfd02d5a4-model-cache" (OuterVolumeSpecName: "model-cache") pod "4358682f-db65-4e80-b47a-741cfd02d5a4" (UID: "4358682f-db65-4e80-b47a-741cfd02d5a4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:17:34.835317 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:34.835296    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4358682f-db65-4e80-b47a-741cfd02d5a4-tmp-dir\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:17:34.835372 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:34.835320    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4358682f-db65-4e80-b47a-741cfd02d5a4-home\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:17:34.835372 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:34.835334    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4358682f-db65-4e80-b47a-741cfd02d5a4-model-cache\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:17:34.837174 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:34.837135    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4358682f-db65-4e80-b47a-741cfd02d5a4-dshm" (OuterVolumeSpecName: "dshm") pod "4358682f-db65-4e80-b47a-741cfd02d5a4" (UID: "4358682f-db65-4e80-b47a-741cfd02d5a4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:17:34.837330 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:34.837303    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4358682f-db65-4e80-b47a-741cfd02d5a4-kube-api-access-j4l65" (OuterVolumeSpecName: "kube-api-access-j4l65") pod "4358682f-db65-4e80-b47a-741cfd02d5a4" (UID: "4358682f-db65-4e80-b47a-741cfd02d5a4"). InnerVolumeSpecName "kube-api-access-j4l65". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:17:34.837400 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:34.837308    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4358682f-db65-4e80-b47a-741cfd02d5a4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4358682f-db65-4e80-b47a-741cfd02d5a4" (UID: "4358682f-db65-4e80-b47a-741cfd02d5a4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:17:34.840255 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:34.840221    2564 generic.go:356] "Generic (PLEG): container finished" podID="4358682f-db65-4e80-b47a-741cfd02d5a4" containerID="c8cb298fca6f310dafe862229ce7c941cf6d8e0de7a4010300b3799acb1ee641" exitCode=0
Sep 18 20:17:34.840377 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:34.840273    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7gqh5" event={"ID":"4358682f-db65-4e80-b47a-741cfd02d5a4","Type":"ContainerDied","Data":"c8cb298fca6f310dafe862229ce7c941cf6d8e0de7a4010300b3799acb1ee641"}
Sep 18 20:17:34.840377 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:34.840330    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7gqh5"
Sep 18 20:17:34.840377 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:34.840350    2564 scope.go:117] "RemoveContainer" containerID="c8cb298fca6f310dafe862229ce7c941cf6d8e0de7a4010300b3799acb1ee641"
Sep 18 20:17:34.840536 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:34.840335    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7gqh5" event={"ID":"4358682f-db65-4e80-b47a-741cfd02d5a4","Type":"ContainerDied","Data":"bf95fcca91dc72625e5eaca9e3ecc0bca516fb083ffa00ce1d455287caa26a63"}
Sep 18 20:17:34.855939 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:34.855916    2564 scope.go:117] "RemoveContainer" containerID="c8cb298fca6f310dafe862229ce7c941cf6d8e0de7a4010300b3799acb1ee641"
Sep 18 20:17:34.856322 ip-10-0-128-161 kubenswrapper[2564]: E0918 20:17:34.856300    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8cb298fca6f310dafe862229ce7c941cf6d8e0de7a4010300b3799acb1ee641\": container with ID starting with c8cb298fca6f310dafe862229ce7c941cf6d8e0de7a4010300b3799acb1ee641 not found: ID does not exist" containerID="c8cb298fca6f310dafe862229ce7c941cf6d8e0de7a4010300b3799acb1ee641"
Sep 18 20:17:34.856395 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:34.856333    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8cb298fca6f310dafe862229ce7c941cf6d8e0de7a4010300b3799acb1ee641"} err="failed to get container status \"c8cb298fca6f310dafe862229ce7c941cf6d8e0de7a4010300b3799acb1ee641\": rpc error: code = NotFound desc = could not find container \"c8cb298fca6f310dafe862229ce7c941cf6d8e0de7a4010300b3799acb1ee641\": container with ID starting with c8cb298fca6f310dafe862229ce7c941cf6d8e0de7a4010300b3799acb1ee641 not found: ID does not exist"
Sep 18 20:17:34.866937 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:34.866903    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7gqh5"]
Sep 18 20:17:34.870678 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:34.870646    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-7gqh5"]
Sep 18 20:17:34.935748 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:34.935714    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4358682f-db65-4e80-b47a-741cfd02d5a4-dshm\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:17:34.935748 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:34.935741    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4l65\" (UniqueName: \"kubernetes.io/projected/4358682f-db65-4e80-b47a-741cfd02d5a4-kube-api-access-j4l65\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:17:34.935748 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:34.935751    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4358682f-db65-4e80-b47a-741cfd02d5a4-tls-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:17:36.000583 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:36.000545    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4358682f-db65-4e80-b47a-741cfd02d5a4" path="/var/lib/kubelet/pods/4358682f-db65-4e80-b47a-741cfd02d5a4/volumes"
Sep 18 20:17:37.632545 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:37.632512    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:43.731933 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:43.731905    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezlnbn"
Sep 18 20:17:46.853785 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:46.853741    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezlnbn"]
Sep 18 20:17:46.867680 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:46.867641    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:46.937738 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:46.937702    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:53.732329 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:53.732292    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezlnbn" podUID="9ccdba2d-cb81-4f9d-9378-c35b93e1474d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:17:56.391715 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:17:56.391672    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:01.883066 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:18:01.883012    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezlnbn" podUID="9ccdba2d-cb81-4f9d-9378-c35b93e1474d" containerName="main" containerID="cri-o://b56f8f42b658ff23a21eac1f1ea8ae361b5cda69c0f5d4fa4b3be190cfc6d6bb" gracePeriod=45
Sep 18 20:18:01.988087 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:18:01.988051    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:02.051336 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:18:02.051295    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:03.731363 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:18:03.731318    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezlnbn" podUID="9ccdba2d-cb81-4f9d-9378-c35b93e1474d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:18:04.091590 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:18:04.091504    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-58vl4_945a444e-9381-4e2f-b4c1-64be0ec88e6c/console-operator/2.log"
Sep 18 20:18:04.097731 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:18:04.097707    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-58vl4_945a444e-9381-4e2f-b4c1-64be0ec88e6c/console-operator/2.log"
Sep 18 20:18:10.542409 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:18:10.542360    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:13.732368 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:18:13.732319    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezlnbn" podUID="9ccdba2d-cb81-4f9d-9378-c35b93e1474d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:18:13.732779 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:18:13.732445    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezlnbn"
Sep 18 20:18:19.081245 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:18:19.081206    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:23.731692 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:18:23.731650    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezlnbn" podUID="9ccdba2d-cb81-4f9d-9378-c35b93e1474d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:18:32.135544 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:18:32.135509    2564 generic.go:356] "Generic (PLEG): container finished" podID="9ccdba2d-cb81-4f9d-9378-c35b93e1474d" containerID="b56f8f42b658ff23a21eac1f1ea8ae361b5cda69c0f5d4fa4b3be190cfc6d6bb" exitCode=0
Sep 18 20:18:32.135867 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:18:32.135583    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezlnbn" event={"ID":"9ccdba2d-cb81-4f9d-9378-c35b93e1474d","Type":"ContainerDied","Data":"b56f8f42b658ff23a21eac1f1ea8ae361b5cda69c0f5d4fa4b3be190cfc6d6bb"}
Sep 18 20:18:32.135867 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:18:32.135640    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezlnbn" event={"ID":"9ccdba2d-cb81-4f9d-9378-c35b93e1474d","Type":"ContainerDied","Data":"05c5fc7ff4cc5ac4c40cacca83ef5d711b54d4e580bc0c3288511144a383d4dd"}
Sep 18 20:18:32.135867 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:18:32.135657    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05c5fc7ff4cc5ac4c40cacca83ef5d711b54d4e580bc0c3288511144a383d4dd"
Sep 18 20:18:32.148641 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:18:32.148560    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezlnbn"
Sep 18 20:18:32.241663 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:18:32.241624    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vg6n6\" (UniqueName: \"kubernetes.io/projected/9ccdba2d-cb81-4f9d-9378-c35b93e1474d-kube-api-access-vg6n6\") pod \"9ccdba2d-cb81-4f9d-9378-c35b93e1474d\" (UID: \"9ccdba2d-cb81-4f9d-9378-c35b93e1474d\") "
Sep 18 20:18:32.241853 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:18:32.241694    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ccdba2d-cb81-4f9d-9378-c35b93e1474d-tls-certs\") pod \"9ccdba2d-cb81-4f9d-9378-c35b93e1474d\" (UID: \"9ccdba2d-cb81-4f9d-9378-c35b93e1474d\") "
Sep 18 20:18:32.244156 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:18:32.244123    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ccdba2d-cb81-4f9d-9378-c35b93e1474d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9ccdba2d-cb81-4f9d-9378-c35b93e1474d" (UID: "9ccdba2d-cb81-4f9d-9378-c35b93e1474d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:18:32.244302 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:18:32.244172    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ccdba2d-cb81-4f9d-9378-c35b93e1474d-kube-api-access-vg6n6" (OuterVolumeSpecName: "kube-api-access-vg6n6") pod "9ccdba2d-cb81-4f9d-9378-c35b93e1474d" (UID: "9ccdba2d-cb81-4f9d-9378-c35b93e1474d"). InnerVolumeSpecName "kube-api-access-vg6n6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:18:32.343213 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:18:32.343165    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg6n6\" (UniqueName: \"kubernetes.io/projected/9ccdba2d-cb81-4f9d-9378-c35b93e1474d-kube-api-access-vg6n6\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:18:32.343213 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:18:32.343211    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ccdba2d-cb81-4f9d-9378-c35b93e1474d-tls-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:18:33.141285 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:18:33.141251    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezlnbn"
Sep 18 20:18:33.165214 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:18:33.165175    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezlnbn"]
Sep 18 20:18:33.170162 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:18:33.170129    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezlnbn"]
Sep 18 20:18:33.999998 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:18:33.999959    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ccdba2d-cb81-4f9d-9378-c35b93e1474d" path="/var/lib/kubelet/pods/9ccdba2d-cb81-4f9d-9378-c35b93e1474d/volumes"
Sep 18 20:18:36.587813 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:18:36.587772    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:36.687764 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:18:36.687728    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:42.603382 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:18:42.603343    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:48.271388 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:18:48.271352    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:48.371693 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:18:48.371650    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:57.232761 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:18:57.232715    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:57.529344 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:18:57.529239    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4wpbv"]
Sep 18 20:18:57.529651 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:18:57.529638    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ccdba2d-cb81-4f9d-9378-c35b93e1474d" containerName="main"
Sep 18 20:18:57.529700 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:18:57.529653    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ccdba2d-cb81-4f9d-9378-c35b93e1474d" containerName="main"
Sep 18 20:18:57.529700 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:18:57.529666    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4358682f-db65-4e80-b47a-741cfd02d5a4" containerName="main"
Sep 18 20:18:57.529700 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:18:57.529672    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4358682f-db65-4e80-b47a-741cfd02d5a4" containerName="main"
Sep 18 20:18:57.529793 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:18:57.529727    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4358682f-db65-4e80-b47a-741cfd02d5a4" containerName="main"
Sep 18 20:18:57.529793 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:18:57.529736    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ccdba2d-cb81-4f9d-9378-c35b93e1474d" containerName="main"
Sep 18 20:18:57.532945 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:18:57.532916    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4wpbv"
Sep 18 20:18:57.535268 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:18:57.535236    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Sep 18 20:18:57.535443 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:18:57.535240    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-router-managed-test-llm-5b1e8f15-epp-sa-dockercfg-s4sxq\""
Sep 18 20:18:57.535771 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:18:57.535750    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Sep 18 20:18:57.535950 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:18:57.535903    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Sep 18 20:18:57.536049 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:18:57.535998    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-f7bgh\""
Sep 18 20:18:57.546574 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:18:57.546542    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4wpbv"]
Sep 18 20:18:57.570469 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:18:57.570433    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:57.668330 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:18:57.668295    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e22f2b4-a09d-485e-a31a-857c1c3ba075-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4wpbv\" (UID: \"0e22f2b4-a09d-485e-a31a-857c1c3ba075\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4wpbv"
Sep 18 20:18:57.668534 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:18:57.668383    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d6fg\" (UniqueName: \"kubernetes.io/projected/0e22f2b4-a09d-485e-a31a-857c1c3ba075-kube-api-access-4d6fg\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4wpbv\" (UID: \"0e22f2b4-a09d-485e-a31a-857c1c3ba075\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4wpbv"
Sep 18 20:18:57.769497 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:18:57.769454    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e22f2b4-a09d-485e-a31a-857c1c3ba075-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4wpbv\" (UID: \"0e22f2b4-a09d-485e-a31a-857c1c3ba075\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4wpbv"
Sep 18 20:18:57.769692 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:18:57.769559    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d6fg\" (UniqueName: \"kubernetes.io/projected/0e22f2b4-a09d-485e-a31a-857c1c3ba075-kube-api-access-4d6fg\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4wpbv\" (UID: \"0e22f2b4-a09d-485e-a31a-857c1c3ba075\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4wpbv"
Sep 18 20:18:57.772127 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:18:57.772100    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e22f2b4-a09d-485e-a31a-857c1c3ba075-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4wpbv\" (UID: \"0e22f2b4-a09d-485e-a31a-857c1c3ba075\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4wpbv"
Sep 18 20:18:57.778514 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:18:57.778471    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d6fg\" (UniqueName: \"kubernetes.io/projected/0e22f2b4-a09d-485e-a31a-857c1c3ba075-kube-api-access-4d6fg\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4wpbv\" (UID: \"0e22f2b4-a09d-485e-a31a-857c1c3ba075\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4wpbv"
Sep 18 20:18:57.846809 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:18:57.846720    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4wpbv"
Sep 18 20:18:58.126636 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:18:58.126608    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4wpbv"]
Sep 18 20:18:58.129298 ip-10-0-128-161 kubenswrapper[2564]: W0918 20:18:58.129267    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e22f2b4_a09d_485e_a31a_857c1c3ba075.slice/crio-c3e0a4a1c34a9c7cad298303c6869761f33c2f20e89c4af48280527793311d5d WatchSource:0}: Error finding container c3e0a4a1c34a9c7cad298303c6869761f33c2f20e89c4af48280527793311d5d: Status 404 returned error can't find the container with id c3e0a4a1c34a9c7cad298303c6869761f33c2f20e89c4af48280527793311d5d
Sep 18 20:18:58.270063 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:18:58.270028    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4wpbv" event={"ID":"0e22f2b4-a09d-485e-a31a-857c1c3ba075","Type":"ContainerStarted","Data":"5f4d77311a94d5f1da963d8887986319e19e2232afc20cc26f9ea5506be901a9"}
Sep 18 20:18:58.270487 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:18:58.270086    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4wpbv" event={"ID":"0e22f2b4-a09d-485e-a31a-857c1c3ba075","Type":"ContainerStarted","Data":"c3e0a4a1c34a9c7cad298303c6869761f33c2f20e89c4af48280527793311d5d"}
Sep 18 20:18:58.270487 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:18:58.270107    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4wpbv"
Sep 18 20:18:58.287816 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:18:58.287760    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4wpbv" podStartSLOduration=1.287742933 podStartE2EDuration="1.287742933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:18:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:18:58.286484439 +0000 UTC m=+2454.796840531" watchObservedRunningTime="2026-09-18 20:18:58.287742933 +0000 UTC m=+2454.798099024"
Sep 18 20:19:05.669549 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:19:05.669512    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:05.801833 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:19:05.801797    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:05.940527 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:19:05.940434    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:08.895573 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:19:08.895531    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:10.697554 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:19:10.697514    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:11.602319 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:19:11.602278    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:13.204177 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:19:13.204138    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:14.212372 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:19:14.212324    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:21.627824 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:19:21.627783    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:29.277650 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:19:29.277614    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4wpbv"
Sep 18 20:19:31.706242 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:19:31.706202    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:32.693856 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:19:32.693802    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:38.352110 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:19:38.352069    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:38.451237 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:19:38.451202    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:20:19.765779 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:20:19.765687    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:20:19.957517 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:20:19.957460    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schen52s4"]
Sep 18 20:20:19.961267 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:20:19.961241    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schen52s4"
Sep 18 20:20:19.965189 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:20:19.965166    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-sa-dockercfg-9vhqc\""
Sep 18 20:20:19.965314 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:20:19.965195    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Sep 18 20:20:19.972680 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:20:19.972653    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schen52s4"]
Sep 18 20:20:19.984455 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:20:19.984420    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:20:20.014802 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:20:20.014765    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8k8f\" (UniqueName: \"kubernetes.io/projected/9fd0890d-c63e-480b-9a47-c2dd2ce655c4-kube-api-access-z8k8f\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schen52s4\" (UID: \"9fd0890d-c63e-480b-9a47-c2dd2ce655c4\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schen52s4"
Sep 18 20:20:20.015011 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:20:20.014819    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fd0890d-c63e-480b-9a47-c2dd2ce655c4-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schen52s4\" (UID: \"9fd0890d-c63e-480b-9a47-c2dd2ce655c4\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schen52s4"
Sep 18 20:20:20.116112 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:20:20.116008    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8k8f\" (UniqueName: \"kubernetes.io/projected/9fd0890d-c63e-480b-9a47-c2dd2ce655c4-kube-api-access-z8k8f\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schen52s4\" (UID: \"9fd0890d-c63e-480b-9a47-c2dd2ce655c4\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schen52s4"
Sep 18 20:20:20.116112 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:20:20.116077    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fd0890d-c63e-480b-9a47-c2dd2ce655c4-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schen52s4\" (UID: \"9fd0890d-c63e-480b-9a47-c2dd2ce655c4\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schen52s4"
Sep 18 20:20:20.118811 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:20:20.118785    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fd0890d-c63e-480b-9a47-c2dd2ce655c4-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schen52s4\" (UID: \"9fd0890d-c63e-480b-9a47-c2dd2ce655c4\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schen52s4"
Sep 18 20:20:20.124436 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:20:20.124412    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8k8f\" (UniqueName: \"kubernetes.io/projected/9fd0890d-c63e-480b-9a47-c2dd2ce655c4-kube-api-access-z8k8f\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schen52s4\" (UID: \"9fd0890d-c63e-480b-9a47-c2dd2ce655c4\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schen52s4"
Sep 18 20:20:20.273788 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:20:20.273739    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schen52s4"
Sep 18 20:20:20.551410 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:20:20.551380    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schen52s4"]
Sep 18 20:20:20.554247 ip-10-0-128-161 kubenswrapper[2564]: W0918 20:20:20.554200    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fd0890d_c63e_480b_9a47_c2dd2ce655c4.slice/crio-ab8f78adb0854a94d1cd6fde0e02266fa345a7f56a58313086de4d31ed8b295f WatchSource:0}: Error finding container ab8f78adb0854a94d1cd6fde0e02266fa345a7f56a58313086de4d31ed8b295f: Status 404 returned error can't find the container with id ab8f78adb0854a94d1cd6fde0e02266fa345a7f56a58313086de4d31ed8b295f
Sep 18 20:20:20.680840 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:20:20.680798    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schen52s4" event={"ID":"9fd0890d-c63e-480b-9a47-c2dd2ce655c4","Type":"ContainerStarted","Data":"d9e306bcc4eb371fde5564b67d524bd963d4558d713a01237ef75420ff55f2c6"}
Sep 18 20:20:20.681095 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:20:20.680867    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schen52s4" event={"ID":"9fd0890d-c63e-480b-9a47-c2dd2ce655c4","Type":"ContainerStarted","Data":"ab8f78adb0854a94d1cd6fde0e02266fa345a7f56a58313086de4d31ed8b295f"}
Sep 18 20:20:20.681095 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:20:20.680898    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schen52s4"
Sep 18 20:20:20.699083 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:20:20.699027    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schen52s4" podStartSLOduration=1.69900681 podStartE2EDuration="1.69900681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:20:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:20:20.696876997 +0000 UTC m=+2537.207233111" watchObservedRunningTime="2026-09-18 20:20:20.69900681 +0000 UTC m=+2537.209362903"
Sep 18 20:20:42.938753 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:20:42.938693    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:20:45.393075 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:20:45.393033    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:20:51.687601 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:20:51.687570    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schen52s4"
Sep 18 20:20:53.081640 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:20:53.081600    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schen52s4"]
Sep 18 20:20:53.084900 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:20:53.084861    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:20:53.148798 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:20:53.148747    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:20:59.609338 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:20:59.609294    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:01.446541 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:21:01.446502    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:01.529586 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:21:01.529547    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:01.629072 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:21:01.629037    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:01.686791 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:21:01.686752    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schen52s4" podUID="9fd0890d-c63e-480b-9a47-c2dd2ce655c4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:21:01.893191 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:21:01.893156    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:06.516008 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:21:06.515940    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:08.103138 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:21:08.103092    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schen52s4" podUID="9fd0890d-c63e-480b-9a47-c2dd2ce655c4" containerName="main" containerID="cri-o://d9e306bcc4eb371fde5564b67d524bd963d4558d713a01237ef75420ff55f2c6" gracePeriod=45
Sep 18 20:21:10.164531 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:21:10.164485    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:11.687278 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:21:11.687242    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schen52s4" podUID="9fd0890d-c63e-480b-9a47-c2dd2ce655c4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:21:20.101159 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:21:20.101116    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:21.686756 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:21:21.686717    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schen52s4" podUID="9fd0890d-c63e-480b-9a47-c2dd2ce655c4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:21:21.687178 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:21:21.686836    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schen52s4"
Sep 18 20:21:25.681304 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:21:25.681257    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:31.687043 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:21:31.686960    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schen52s4" podUID="9fd0890d-c63e-480b-9a47-c2dd2ce655c4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:21:38.351213 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:21:38.351187    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schen52s4"
Sep 18 20:21:38.429797 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:21:38.429758    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z8k8f\" (UniqueName: \"kubernetes.io/projected/9fd0890d-c63e-480b-9a47-c2dd2ce655c4-kube-api-access-z8k8f\") pod \"9fd0890d-c63e-480b-9a47-c2dd2ce655c4\" (UID: \"9fd0890d-c63e-480b-9a47-c2dd2ce655c4\") "
Sep 18 20:21:38.430023 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:21:38.429825    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fd0890d-c63e-480b-9a47-c2dd2ce655c4-tls-certs\") pod \"9fd0890d-c63e-480b-9a47-c2dd2ce655c4\" (UID: \"9fd0890d-c63e-480b-9a47-c2dd2ce655c4\") "
Sep 18 20:21:38.432154 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:21:38.432126    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fd0890d-c63e-480b-9a47-c2dd2ce655c4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9fd0890d-c63e-480b-9a47-c2dd2ce655c4" (UID: "9fd0890d-c63e-480b-9a47-c2dd2ce655c4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:21:38.432264 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:21:38.432172    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fd0890d-c63e-480b-9a47-c2dd2ce655c4-kube-api-access-z8k8f" (OuterVolumeSpecName: "kube-api-access-z8k8f") pod "9fd0890d-c63e-480b-9a47-c2dd2ce655c4" (UID: "9fd0890d-c63e-480b-9a47-c2dd2ce655c4"). InnerVolumeSpecName "kube-api-access-z8k8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:21:38.531051 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:21:38.531008    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8k8f\" (UniqueName: \"kubernetes.io/projected/9fd0890d-c63e-480b-9a47-c2dd2ce655c4-kube-api-access-z8k8f\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:21:38.531051 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:21:38.531038    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fd0890d-c63e-480b-9a47-c2dd2ce655c4-tls-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:21:39.075027 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:21:39.073051    2564 generic.go:356] "Generic (PLEG): container finished" podID="9fd0890d-c63e-480b-9a47-c2dd2ce655c4" containerID="d9e306bcc4eb371fde5564b67d524bd963d4558d713a01237ef75420ff55f2c6" exitCode=0
Sep 18 20:21:39.075027 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:21:39.073201    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schen52s4" event={"ID":"9fd0890d-c63e-480b-9a47-c2dd2ce655c4","Type":"ContainerDied","Data":"d9e306bcc4eb371fde5564b67d524bd963d4558d713a01237ef75420ff55f2c6"}
Sep 18 20:21:39.075027 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:21:39.073248    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schen52s4" event={"ID":"9fd0890d-c63e-480b-9a47-c2dd2ce655c4","Type":"ContainerDied","Data":"ab8f78adb0854a94d1cd6fde0e02266fa345a7f56a58313086de4d31ed8b295f"}
Sep 18 20:21:39.075027 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:21:39.073275    2564 scope.go:117] "RemoveContainer" containerID="d9e306bcc4eb371fde5564b67d524bd963d4558d713a01237ef75420ff55f2c6"
Sep 18 20:21:39.075027 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:21:39.073453    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schen52s4"
Sep 18 20:21:39.084029 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:21:39.084010    2564 scope.go:117] "RemoveContainer" containerID="d9e306bcc4eb371fde5564b67d524bd963d4558d713a01237ef75420ff55f2c6"
Sep 18 20:21:39.084322 ip-10-0-128-161 kubenswrapper[2564]: E0918 20:21:39.084302    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9e306bcc4eb371fde5564b67d524bd963d4558d713a01237ef75420ff55f2c6\": container with ID starting with d9e306bcc4eb371fde5564b67d524bd963d4558d713a01237ef75420ff55f2c6 not found: ID does not exist" containerID="d9e306bcc4eb371fde5564b67d524bd963d4558d713a01237ef75420ff55f2c6"
Sep 18 20:21:39.084377 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:21:39.084331    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9e306bcc4eb371fde5564b67d524bd963d4558d713a01237ef75420ff55f2c6"} err="failed to get container status \"d9e306bcc4eb371fde5564b67d524bd963d4558d713a01237ef75420ff55f2c6\": rpc error: code = NotFound desc = could not find container \"d9e306bcc4eb371fde5564b67d524bd963d4558d713a01237ef75420ff55f2c6\": container with ID starting with d9e306bcc4eb371fde5564b67d524bd963d4558d713a01237ef75420ff55f2c6 not found: ID does not exist"
Sep 18 20:21:39.096394 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:21:39.096356    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schen52s4"]
Sep 18 20:21:39.100684 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:21:39.100652    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schen52s4"]
Sep 18 20:21:40.000966 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:21:40.000925    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9fd0890d-c63e-480b-9a47-c2dd2ce655c4" path="/var/lib/kubelet/pods/9fd0890d-c63e-480b-9a47-c2dd2ce655c4/volumes"
Sep 18 20:21:42.435766 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:21:42.435724    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:44.898105 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:21:44.898059    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:46.015460 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:21:46.015425    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4wpbv"]
Sep 18 20:21:46.017199 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:21:46.017167    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:46.098049 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:21:46.098016    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:49.278519 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:21:49.278476    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4wpbv" podUID="0e22f2b4-a09d-485e-a31a-857c1c3ba075" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:21:59.278357 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:21:59.278319    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4wpbv" podUID="0e22f2b4-a09d-485e-a31a-857c1c3ba075" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:22:01.035898 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:01.035850    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4wpbv" podUID="0e22f2b4-a09d-485e-a31a-857c1c3ba075" containerName="main" containerID="cri-o://5f4d77311a94d5f1da963d8887986319e19e2232afc20cc26f9ea5506be901a9" gracePeriod=45
Sep 18 20:22:09.277760 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:09.277711    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4wpbv" podUID="0e22f2b4-a09d-485e-a31a-857c1c3ba075" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:22:09.278310 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:09.277849    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4wpbv"
Sep 18 20:22:19.278307 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:19.278264    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4wpbv" podUID="0e22f2b4-a09d-485e-a31a-857c1c3ba075" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:22:29.277873 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:29.277825    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4wpbv" podUID="0e22f2b4-a09d-485e-a31a-857c1c3ba075" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:22:31.301364 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:31.301290    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4wpbv"
Sep 18 20:22:31.314785 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:31.314749    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e22f2b4-a09d-485e-a31a-857c1c3ba075-tls-certs\") pod \"0e22f2b4-a09d-485e-a31a-857c1c3ba075\" (UID: \"0e22f2b4-a09d-485e-a31a-857c1c3ba075\") "
Sep 18 20:22:31.314975 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:31.314913    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4d6fg\" (UniqueName: \"kubernetes.io/projected/0e22f2b4-a09d-485e-a31a-857c1c3ba075-kube-api-access-4d6fg\") pod \"0e22f2b4-a09d-485e-a31a-857c1c3ba075\" (UID: \"0e22f2b4-a09d-485e-a31a-857c1c3ba075\") "
Sep 18 20:22:31.316379 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:31.316276    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:31.317969 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:31.317929    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e22f2b4-a09d-485e-a31a-857c1c3ba075-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0e22f2b4-a09d-485e-a31a-857c1c3ba075" (UID: "0e22f2b4-a09d-485e-a31a-857c1c3ba075"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:22:31.318480 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:31.318455    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e22f2b4-a09d-485e-a31a-857c1c3ba075-kube-api-access-4d6fg" (OuterVolumeSpecName: "kube-api-access-4d6fg") pod "0e22f2b4-a09d-485e-a31a-857c1c3ba075" (UID: "0e22f2b4-a09d-485e-a31a-857c1c3ba075"). InnerVolumeSpecName "kube-api-access-4d6fg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:22:31.330165 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:31.330137    2564 generic.go:356] "Generic (PLEG): container finished" podID="0e22f2b4-a09d-485e-a31a-857c1c3ba075" containerID="5f4d77311a94d5f1da963d8887986319e19e2232afc20cc26f9ea5506be901a9" exitCode=0
Sep 18 20:22:31.330293 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:31.330186    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4wpbv"
Sep 18 20:22:31.330293 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:31.330214    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4wpbv" event={"ID":"0e22f2b4-a09d-485e-a31a-857c1c3ba075","Type":"ContainerDied","Data":"5f4d77311a94d5f1da963d8887986319e19e2232afc20cc26f9ea5506be901a9"}
Sep 18 20:22:31.330293 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:31.330288    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4wpbv" event={"ID":"0e22f2b4-a09d-485e-a31a-857c1c3ba075","Type":"ContainerDied","Data":"c3e0a4a1c34a9c7cad298303c6869761f33c2f20e89c4af48280527793311d5d"}
Sep 18 20:22:31.330396 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:31.330313    2564 scope.go:117] "RemoveContainer" containerID="5f4d77311a94d5f1da963d8887986319e19e2232afc20cc26f9ea5506be901a9"
Sep 18 20:22:31.340228 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:31.340208    2564 scope.go:117] "RemoveContainer" containerID="5f4d77311a94d5f1da963d8887986319e19e2232afc20cc26f9ea5506be901a9"
Sep 18 20:22:31.340537 ip-10-0-128-161 kubenswrapper[2564]: E0918 20:22:31.340516    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f4d77311a94d5f1da963d8887986319e19e2232afc20cc26f9ea5506be901a9\": container with ID starting with 5f4d77311a94d5f1da963d8887986319e19e2232afc20cc26f9ea5506be901a9 not found: ID does not exist" containerID="5f4d77311a94d5f1da963d8887986319e19e2232afc20cc26f9ea5506be901a9"
Sep 18 20:22:31.340620 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:31.340544    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f4d77311a94d5f1da963d8887986319e19e2232afc20cc26f9ea5506be901a9"} err="failed to get container status \"5f4d77311a94d5f1da963d8887986319e19e2232afc20cc26f9ea5506be901a9\": rpc error: code = NotFound desc = could not find container \"5f4d77311a94d5f1da963d8887986319e19e2232afc20cc26f9ea5506be901a9\": container with ID starting with 5f4d77311a94d5f1da963d8887986319e19e2232afc20cc26f9ea5506be901a9 not found: ID does not exist"
Sep 18 20:22:31.353507 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:31.353476    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4wpbv"]
Sep 18 20:22:31.363500 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:31.363471    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche4wpbv"]
Sep 18 20:22:31.416034 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:31.415974    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e22f2b4-a09d-485e-a31a-857c1c3ba075-tls-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:22:31.416034 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:31.416026    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4d6fg\" (UniqueName: \"kubernetes.io/projected/0e22f2b4-a09d-485e-a31a-857c1c3ba075-kube-api-access-4d6fg\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:22:31.420000 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:31.419959    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:31.528747 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:31.528710    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:31.852845 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:31.852808    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:32.001126 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:32.001090    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e22f2b4-a09d-485e-a31a-857c1c3ba075" path="/var/lib/kubelet/pods/0e22f2b4-a09d-485e-a31a-857c1c3ba075/volumes"
Sep 18 20:22:32.846151 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:32.846112    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:33.592210 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:33.592179    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:33.693784 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:33.693748    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:36.393819 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:36.393780    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:39.846835 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:39.846789    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:45.663626 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:45.663576    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:45.763556 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:45.763520    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:48.310863 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:48.310801    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-dm5lc"]
Sep 18 20:22:48.311307 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:48.311237    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e22f2b4-a09d-485e-a31a-857c1c3ba075" containerName="main"
Sep 18 20:22:48.311307 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:48.311253    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e22f2b4-a09d-485e-a31a-857c1c3ba075" containerName="main"
Sep 18 20:22:48.311307 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:48.311283    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fd0890d-c63e-480b-9a47-c2dd2ce655c4" containerName="main"
Sep 18 20:22:48.311307 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:48.311291    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fd0890d-c63e-480b-9a47-c2dd2ce655c4" containerName="main"
Sep 18 20:22:48.311535 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:48.311382    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e22f2b4-a09d-485e-a31a-857c1c3ba075" containerName="main"
Sep 18 20:22:48.311535 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:48.311394    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fd0890d-c63e-480b-9a47-c2dd2ce655c4" containerName="main"
Sep 18 20:22:48.314525 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:48.314502    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-dm5lc"
Sep 18 20:22:48.317587 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:48.317558    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"openshift-service-ca.crt\""
Sep 18 20:22:48.317752 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:48.317590    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"default-dockercfg-hd7fv\""
Sep 18 20:22:48.317752 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:48.317694    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"kube-root-ca.crt\""
Sep 18 20:22:48.317752 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:48.317716    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"owner-v1-kserve-self-signed-certs\""
Sep 18 20:22:48.325264 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:48.325237    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-dm5lc"]
Sep 18 20:22:48.335889 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:48.335772    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:48.363043 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:48.363000    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df773247-0709-461f-8d79-866de7a188c5-tls-certs\") pod \"owner-v1-kserve-75579ff5bc-dm5lc\" (UID: \"df773247-0709-461f-8d79-866de7a188c5\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-dm5lc"
Sep 18 20:22:48.363246 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:48.363053    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df773247-0709-461f-8d79-866de7a188c5-home\") pod \"owner-v1-kserve-75579ff5bc-dm5lc\" (UID: \"df773247-0709-461f-8d79-866de7a188c5\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-dm5lc"
Sep 18 20:22:48.363246 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:48.363162    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pbb4\" (UniqueName: \"kubernetes.io/projected/df773247-0709-461f-8d79-866de7a188c5-kube-api-access-2pbb4\") pod \"owner-v1-kserve-75579ff5bc-dm5lc\" (UID: \"df773247-0709-461f-8d79-866de7a188c5\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-dm5lc"
Sep 18 20:22:48.363246 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:48.363208    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df773247-0709-461f-8d79-866de7a188c5-tmp-dir\") pod \"owner-v1-kserve-75579ff5bc-dm5lc\" (UID: \"df773247-0709-461f-8d79-866de7a188c5\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-dm5lc"
Sep 18 20:22:48.363377 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:48.363299    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df773247-0709-461f-8d79-866de7a188c5-model-cache\") pod \"owner-v1-kserve-75579ff5bc-dm5lc\" (UID: \"df773247-0709-461f-8d79-866de7a188c5\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-dm5lc"
Sep 18 20:22:48.363377 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:48.363320    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df773247-0709-461f-8d79-866de7a188c5-dshm\") pod \"owner-v1-kserve-75579ff5bc-dm5lc\" (UID: \"df773247-0709-461f-8d79-866de7a188c5\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-dm5lc"
Sep 18 20:22:48.464772 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:48.464732    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pbb4\" (UniqueName: \"kubernetes.io/projected/df773247-0709-461f-8d79-866de7a188c5-kube-api-access-2pbb4\") pod \"owner-v1-kserve-75579ff5bc-dm5lc\" (UID: \"df773247-0709-461f-8d79-866de7a188c5\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-dm5lc"
Sep 18 20:22:48.464772 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:48.464781    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df773247-0709-461f-8d79-866de7a188c5-tmp-dir\") pod \"owner-v1-kserve-75579ff5bc-dm5lc\" (UID: \"df773247-0709-461f-8d79-866de7a188c5\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-dm5lc"
Sep 18 20:22:48.465107 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:48.464838    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df773247-0709-461f-8d79-866de7a188c5-model-cache\") pod \"owner-v1-kserve-75579ff5bc-dm5lc\" (UID: \"df773247-0709-461f-8d79-866de7a188c5\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-dm5lc"
Sep 18 20:22:48.465107 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:48.464856    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df773247-0709-461f-8d79-866de7a188c5-dshm\") pod \"owner-v1-kserve-75579ff5bc-dm5lc\" (UID: \"df773247-0709-461f-8d79-866de7a188c5\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-dm5lc"
Sep 18 20:22:48.465107 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:48.464877    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df773247-0709-461f-8d79-866de7a188c5-tls-certs\") pod \"owner-v1-kserve-75579ff5bc-dm5lc\" (UID: \"df773247-0709-461f-8d79-866de7a188c5\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-dm5lc"
Sep 18 20:22:48.465107 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:48.464916    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df773247-0709-461f-8d79-866de7a188c5-home\") pod \"owner-v1-kserve-75579ff5bc-dm5lc\" (UID: \"df773247-0709-461f-8d79-866de7a188c5\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-dm5lc"
Sep 18 20:22:48.465290 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:48.465275    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df773247-0709-461f-8d79-866de7a188c5-tmp-dir\") pod \"owner-v1-kserve-75579ff5bc-dm5lc\" (UID: \"df773247-0709-461f-8d79-866de7a188c5\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-dm5lc"
Sep 18 20:22:48.465340 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:48.465317    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df773247-0709-461f-8d79-866de7a188c5-model-cache\") pod \"owner-v1-kserve-75579ff5bc-dm5lc\" (UID: \"df773247-0709-461f-8d79-866de7a188c5\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-dm5lc"
Sep 18 20:22:48.465390 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:48.465378    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df773247-0709-461f-8d79-866de7a188c5-home\") pod \"owner-v1-kserve-75579ff5bc-dm5lc\" (UID: \"df773247-0709-461f-8d79-866de7a188c5\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-dm5lc"
Sep 18 20:22:48.467360 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:48.467337    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df773247-0709-461f-8d79-866de7a188c5-dshm\") pod \"owner-v1-kserve-75579ff5bc-dm5lc\" (UID: \"df773247-0709-461f-8d79-866de7a188c5\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-dm5lc"
Sep 18 20:22:48.467602 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:48.467584    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df773247-0709-461f-8d79-866de7a188c5-tls-certs\") pod \"owner-v1-kserve-75579ff5bc-dm5lc\" (UID: \"df773247-0709-461f-8d79-866de7a188c5\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-dm5lc"
Sep 18 20:22:48.473413 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:48.473386    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pbb4\" (UniqueName: \"kubernetes.io/projected/df773247-0709-461f-8d79-866de7a188c5-kube-api-access-2pbb4\") pod \"owner-v1-kserve-75579ff5bc-dm5lc\" (UID: \"df773247-0709-461f-8d79-866de7a188c5\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-dm5lc"
Sep 18 20:22:48.626914 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:48.626796    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-dm5lc"
Sep 18 20:22:48.916471 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:48.916443    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-dm5lc"]
Sep 18 20:22:48.919012 ip-10-0-128-161 kubenswrapper[2564]: W0918 20:22:48.918963    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf773247_0709_461f_8d79_866de7a188c5.slice/crio-5bb28dd4b177f781d4014a1fa2304d50d74ae2bd4aada4a7ae1d779f55765a99 WatchSource:0}: Error finding container 5bb28dd4b177f781d4014a1fa2304d50d74ae2bd4aada4a7ae1d779f55765a99: Status 404 returned error can't find the container with id 5bb28dd4b177f781d4014a1fa2304d50d74ae2bd4aada4a7ae1d779f55765a99
Sep 18 20:22:49.419282 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:49.419243    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-dm5lc" event={"ID":"df773247-0709-461f-8d79-866de7a188c5","Type":"ContainerStarted","Data":"6bf09c0fe290c01d3d080652a5068f5fcd56274b100cd01f3e570943427252a5"}
Sep 18 20:22:49.419736 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:49.419305    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-dm5lc" event={"ID":"df773247-0709-461f-8d79-866de7a188c5","Type":"ContainerStarted","Data":"5bb28dd4b177f781d4014a1fa2304d50d74ae2bd4aada4a7ae1d779f55765a99"}
Sep 18 20:22:49.438241 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:49.438177    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-dm5lc" podStartSLOduration=1.438153912 podStartE2EDuration="1.438153912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:22:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:22:49.437728939 +0000 UTC m=+2685.948085029" watchObservedRunningTime="2026-09-18 20:22:49.438153912 +0000 UTC m=+2685.948510009"
Sep 18 20:22:49.627760 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:49.627714    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-dm5lc"
Sep 18 20:22:51.940592 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:51.940554    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:58.627757 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:58.627640    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-dm5lc"
Sep 18 20:22:58.640712 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:58.640682    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-dm5lc"
Sep 18 20:22:59.480235 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:22:59.480206    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-dm5lc"
Sep 18 20:23:02.239375 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:02.239326    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:23:02.416016 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:02.415956    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9gn8q"]
Sep 18 20:23:02.419623 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:02.419584    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9gn8q"
Sep 18 20:23:02.422135 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:02.422109    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 18 20:23:02.422282 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:02.422231    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 18 20:23:02.422282 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:02.422231    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Sep 18 20:23:02.422396 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:02.422233    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-router-managed-test-llm-4b931143-epp-sa-dockercfg-ftvnj\""
Sep 18 20:23:02.422396 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:02.422302    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-ckdwr\""
Sep 18 20:23:02.427615 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:02.427374    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9gn8q"]
Sep 18 20:23:02.440002 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:02.439957    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:23:02.490459 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:02.490372    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91411429-61a5-42bc-bf80-e87a2b47b68f-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9gn8q\" (UID: \"91411429-61a5-42bc-bf80-e87a2b47b68f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9gn8q"
Sep 18 20:23:02.490607 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:02.490499    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27z4h\" (UniqueName: \"kubernetes.io/projected/91411429-61a5-42bc-bf80-e87a2b47b68f-kube-api-access-27z4h\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9gn8q\" (UID: \"91411429-61a5-42bc-bf80-e87a2b47b68f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9gn8q"
Sep 18 20:23:02.591165 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:02.591124    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27z4h\" (UniqueName: \"kubernetes.io/projected/91411429-61a5-42bc-bf80-e87a2b47b68f-kube-api-access-27z4h\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9gn8q\" (UID: \"91411429-61a5-42bc-bf80-e87a2b47b68f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9gn8q"
Sep 18 20:23:02.591374 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:02.591186    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91411429-61a5-42bc-bf80-e87a2b47b68f-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9gn8q\" (UID: \"91411429-61a5-42bc-bf80-e87a2b47b68f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9gn8q"
Sep 18 20:23:02.593802 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:02.593775    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91411429-61a5-42bc-bf80-e87a2b47b68f-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9gn8q\" (UID: \"91411429-61a5-42bc-bf80-e87a2b47b68f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9gn8q"
Sep 18 20:23:02.599558 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:02.599527    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27z4h\" (UniqueName: \"kubernetes.io/projected/91411429-61a5-42bc-bf80-e87a2b47b68f-kube-api-access-27z4h\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9gn8q\" (UID: \"91411429-61a5-42bc-bf80-e87a2b47b68f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9gn8q"
Sep 18 20:23:02.731556 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:02.731516    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9gn8q"
Sep 18 20:23:03.005406 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:03.005176    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9gn8q"]
Sep 18 20:23:03.008300 ip-10-0-128-161 kubenswrapper[2564]: W0918 20:23:03.008271    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91411429_61a5_42bc_bf80_e87a2b47b68f.slice/crio-8dd566af291434b73fcfd9a275d0e74acb9d1c2aafb01249367317e956f5e193 WatchSource:0}: Error finding container 8dd566af291434b73fcfd9a275d0e74acb9d1c2aafb01249367317e956f5e193: Status 404 returned error can't find the container with id 8dd566af291434b73fcfd9a275d0e74acb9d1c2aafb01249367317e956f5e193
Sep 18 20:23:03.497744 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:03.497706    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9gn8q" event={"ID":"91411429-61a5-42bc-bf80-e87a2b47b68f","Type":"ContainerStarted","Data":"76320f935d5083620377e8e54b3fa5e3db64246ea582fd7f51cf3ec04ef18b91"}
Sep 18 20:23:03.498208 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:03.497770    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9gn8q" event={"ID":"91411429-61a5-42bc-bf80-e87a2b47b68f","Type":"ContainerStarted","Data":"8dd566af291434b73fcfd9a275d0e74acb9d1c2aafb01249367317e956f5e193"}
Sep 18 20:23:03.498208 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:03.497801    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9gn8q"
Sep 18 20:23:03.513809 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:03.513737    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9gn8q" podStartSLOduration=1.5136981139999999 podStartE2EDuration="1.513698114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:23:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:23:03.512277449 +0000 UTC m=+2700.022633559" watchObservedRunningTime="2026-09-18 20:23:03.513698114 +0000 UTC m=+2700.024054205"
Sep 18 20:23:04.120938 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:04.120908    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-58vl4_945a444e-9381-4e2f-b4c1-64be0ec88e6c/console-operator/2.log"
Sep 18 20:23:04.128617 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:04.128588    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-58vl4_945a444e-9381-4e2f-b4c1-64be0ec88e6c/console-operator/2.log"
Sep 18 20:23:13.469008 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:13.468955    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:23:19.175352 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:19.175315    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:23:19.243591 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:19.243546    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:23:30.731418 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:30.731380    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:23:34.506326 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:34.506293    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9gn8q"
Sep 18 20:23:38.831439 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:38.831399    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:23:38.993685 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:38.993644    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche72hq6"]
Sep 18 20:23:38.998085 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:38.998062    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche72hq6"
Sep 18 20:23:39.000653 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:39.000628    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Sep 18 20:23:39.000853 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:39.000629    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-sa-dockercfg-fzmgx\""
Sep 18 20:23:39.007215 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:39.007188    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche72hq6"]
Sep 18 20:23:39.021874 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:39.021840    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:23:39.111305 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:39.111204    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6q58\" (UniqueName: \"kubernetes.io/projected/c7137477-bed9-4efc-bd06-57dcb8e632ce-kube-api-access-r6q58\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche72hq6\" (UID: \"c7137477-bed9-4efc-bd06-57dcb8e632ce\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche72hq6"
Sep 18 20:23:39.111305 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:39.111275    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7137477-bed9-4efc-bd06-57dcb8e632ce-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche72hq6\" (UID: \"c7137477-bed9-4efc-bd06-57dcb8e632ce\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche72hq6"
Sep 18 20:23:39.212435 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:39.212395    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6q58\" (UniqueName: \"kubernetes.io/projected/c7137477-bed9-4efc-bd06-57dcb8e632ce-kube-api-access-r6q58\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche72hq6\" (UID: \"c7137477-bed9-4efc-bd06-57dcb8e632ce\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche72hq6"
Sep 18 20:23:39.212662 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:39.212452    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7137477-bed9-4efc-bd06-57dcb8e632ce-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche72hq6\" (UID: \"c7137477-bed9-4efc-bd06-57dcb8e632ce\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche72hq6"
Sep 18 20:23:39.215165 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:39.215137    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7137477-bed9-4efc-bd06-57dcb8e632ce-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche72hq6\" (UID: \"c7137477-bed9-4efc-bd06-57dcb8e632ce\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche72hq6"
Sep 18 20:23:39.221846 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:39.221817    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6q58\" (UniqueName: \"kubernetes.io/projected/c7137477-bed9-4efc-bd06-57dcb8e632ce-kube-api-access-r6q58\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche72hq6\" (UID: \"c7137477-bed9-4efc-bd06-57dcb8e632ce\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche72hq6"
Sep 18 20:23:39.311093 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:39.311055    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche72hq6"
Sep 18 20:23:39.585162 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:39.585135    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche72hq6"]
Sep 18 20:23:39.587605 ip-10-0-128-161 kubenswrapper[2564]: W0918 20:23:39.587575    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7137477_bed9_4efc_bd06_57dcb8e632ce.slice/crio-cfb50717a6bd97e15da6bfc7e7aa6c4b5fcadb42abf058618e45084e6e0cc454 WatchSource:0}: Error finding container cfb50717a6bd97e15da6bfc7e7aa6c4b5fcadb42abf058618e45084e6e0cc454: Status 404 returned error can't find the container with id cfb50717a6bd97e15da6bfc7e7aa6c4b5fcadb42abf058618e45084e6e0cc454
Sep 18 20:23:39.678894 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:39.678858    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche72hq6" event={"ID":"c7137477-bed9-4efc-bd06-57dcb8e632ce","Type":"ContainerStarted","Data":"07fec5689a3823fc135a8daa93336e023f353cd20a4f026106f8ba121ac2e17d"}
Sep 18 20:23:39.679086 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:39.678910    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche72hq6" event={"ID":"c7137477-bed9-4efc-bd06-57dcb8e632ce","Type":"ContainerStarted","Data":"cfb50717a6bd97e15da6bfc7e7aa6c4b5fcadb42abf058618e45084e6e0cc454"}
Sep 18 20:23:39.679086 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:39.678967    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche72hq6"
Sep 18 20:23:39.701331 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:39.701281    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche72hq6" podStartSLOduration=1.701266052 podStartE2EDuration="1.701266052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:23:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:23:39.698516344 +0000 UTC m=+2736.208872472" watchObservedRunningTime="2026-09-18 20:23:39.701266052 +0000 UTC m=+2736.211622142"
Sep 18 20:23:42.780661 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:42.780625    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-dm5lc"]
Sep 18 20:23:42.808228 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:42.808181    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:23:53.707861 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:53.707826    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:23:57.804109 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:57.804035    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-dm5lc" podUID="df773247-0709-461f-8d79-866de7a188c5" containerName="main" containerID="cri-o://6bf09c0fe290c01d3d080652a5068f5fcd56274b100cd01f3e570943427252a5" gracePeriod=45
Sep 18 20:23:58.065812 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:58.065747    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-dm5lc"
Sep 18 20:23:58.185693 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:58.185657    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df773247-0709-461f-8d79-866de7a188c5-dshm\") pod \"df773247-0709-461f-8d79-866de7a188c5\" (UID: \"df773247-0709-461f-8d79-866de7a188c5\") "
Sep 18 20:23:58.185894 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:58.185755    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2pbb4\" (UniqueName: \"kubernetes.io/projected/df773247-0709-461f-8d79-866de7a188c5-kube-api-access-2pbb4\") pod \"df773247-0709-461f-8d79-866de7a188c5\" (UID: \"df773247-0709-461f-8d79-866de7a188c5\") "
Sep 18 20:23:58.185894 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:58.185785    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df773247-0709-461f-8d79-866de7a188c5-home\") pod \"df773247-0709-461f-8d79-866de7a188c5\" (UID: \"df773247-0709-461f-8d79-866de7a188c5\") "
Sep 18 20:23:58.185894 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:58.185814    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df773247-0709-461f-8d79-866de7a188c5-model-cache\") pod \"df773247-0709-461f-8d79-866de7a188c5\" (UID: \"df773247-0709-461f-8d79-866de7a188c5\") "
Sep 18 20:23:58.185894 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:58.185877    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df773247-0709-461f-8d79-866de7a188c5-tmp-dir\") pod \"df773247-0709-461f-8d79-866de7a188c5\" (UID: \"df773247-0709-461f-8d79-866de7a188c5\") "
Sep 18 20:23:58.186159 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:58.185917    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df773247-0709-461f-8d79-866de7a188c5-tls-certs\") pod \"df773247-0709-461f-8d79-866de7a188c5\" (UID: \"df773247-0709-461f-8d79-866de7a188c5\") "
Sep 18 20:23:58.186159 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:58.186099    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df773247-0709-461f-8d79-866de7a188c5-home" (OuterVolumeSpecName: "home") pod "df773247-0709-461f-8d79-866de7a188c5" (UID: "df773247-0709-461f-8d79-866de7a188c5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:23:58.186247 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:58.186162    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df773247-0709-461f-8d79-866de7a188c5-model-cache" (OuterVolumeSpecName: "model-cache") pod "df773247-0709-461f-8d79-866de7a188c5" (UID: "df773247-0709-461f-8d79-866de7a188c5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:23:58.186313 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:58.186293    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df773247-0709-461f-8d79-866de7a188c5-home\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:23:58.186356 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:58.186325    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df773247-0709-461f-8d79-866de7a188c5-model-cache\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:23:58.186356 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:58.186282    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df773247-0709-461f-8d79-866de7a188c5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "df773247-0709-461f-8d79-866de7a188c5" (UID: "df773247-0709-461f-8d79-866de7a188c5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:23:58.188189 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:58.188168    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df773247-0709-461f-8d79-866de7a188c5-dshm" (OuterVolumeSpecName: "dshm") pod "df773247-0709-461f-8d79-866de7a188c5" (UID: "df773247-0709-461f-8d79-866de7a188c5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:23:58.188275 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:58.188211    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df773247-0709-461f-8d79-866de7a188c5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "df773247-0709-461f-8d79-866de7a188c5" (UID: "df773247-0709-461f-8d79-866de7a188c5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:23:58.188319 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:58.188265    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df773247-0709-461f-8d79-866de7a188c5-kube-api-access-2pbb4" (OuterVolumeSpecName: "kube-api-access-2pbb4") pod "df773247-0709-461f-8d79-866de7a188c5" (UID: "df773247-0709-461f-8d79-866de7a188c5"). InnerVolumeSpecName "kube-api-access-2pbb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:23:58.287398 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:58.287358    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pbb4\" (UniqueName: \"kubernetes.io/projected/df773247-0709-461f-8d79-866de7a188c5-kube-api-access-2pbb4\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:23:58.287398 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:58.287394    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df773247-0709-461f-8d79-866de7a188c5-tmp-dir\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:23:58.287398 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:58.287406    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df773247-0709-461f-8d79-866de7a188c5-tls-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:23:58.287645 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:58.287416    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df773247-0709-461f-8d79-866de7a188c5-dshm\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:23:58.785976 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:58.785943    2564 generic.go:356] "Generic (PLEG): container finished" podID="df773247-0709-461f-8d79-866de7a188c5" containerID="6bf09c0fe290c01d3d080652a5068f5fcd56274b100cd01f3e570943427252a5" exitCode=0
Sep 18 20:23:58.786201 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:58.786015    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-dm5lc"
Sep 18 20:23:58.786201 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:58.786020    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-dm5lc" event={"ID":"df773247-0709-461f-8d79-866de7a188c5","Type":"ContainerDied","Data":"6bf09c0fe290c01d3d080652a5068f5fcd56274b100cd01f3e570943427252a5"}
Sep 18 20:23:58.786201 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:58.786075    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-dm5lc" event={"ID":"df773247-0709-461f-8d79-866de7a188c5","Type":"ContainerDied","Data":"5bb28dd4b177f781d4014a1fa2304d50d74ae2bd4aada4a7ae1d779f55765a99"}
Sep 18 20:23:58.786201 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:58.786096    2564 scope.go:117] "RemoveContainer" containerID="6bf09c0fe290c01d3d080652a5068f5fcd56274b100cd01f3e570943427252a5"
Sep 18 20:23:58.795530 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:58.795513    2564 scope.go:117] "RemoveContainer" containerID="6bf09c0fe290c01d3d080652a5068f5fcd56274b100cd01f3e570943427252a5"
Sep 18 20:23:58.795858 ip-10-0-128-161 kubenswrapper[2564]: E0918 20:23:58.795835    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bf09c0fe290c01d3d080652a5068f5fcd56274b100cd01f3e570943427252a5\": container with ID starting with 6bf09c0fe290c01d3d080652a5068f5fcd56274b100cd01f3e570943427252a5 not found: ID does not exist" containerID="6bf09c0fe290c01d3d080652a5068f5fcd56274b100cd01f3e570943427252a5"
Sep 18 20:23:58.795934 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:58.795865    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bf09c0fe290c01d3d080652a5068f5fcd56274b100cd01f3e570943427252a5"} err="failed to get container status \"6bf09c0fe290c01d3d080652a5068f5fcd56274b100cd01f3e570943427252a5\": rpc error: code = NotFound desc = could not find container \"6bf09c0fe290c01d3d080652a5068f5fcd56274b100cd01f3e570943427252a5\": container with ID starting with 6bf09c0fe290c01d3d080652a5068f5fcd56274b100cd01f3e570943427252a5 not found: ID does not exist"
Sep 18 20:23:58.808020 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:58.807966    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-dm5lc"]
Sep 18 20:23:58.814118 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:23:58.814089    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-dm5lc"]
Sep 18 20:24:00.000497 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:24:00.000462    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df773247-0709-461f-8d79-866de7a188c5" path="/var/lib/kubelet/pods/df773247-0709-461f-8d79-866de7a188c5/volumes"
Sep 18 20:24:04.793597 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:24:04.793559    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:04.891391 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:24:04.891351    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:05.099033 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:24:05.098920    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:05.282079 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:24:05.282038    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:08.526593 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:24:08.526554    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:09.194869 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:24:09.194837    2564 scope.go:117] "RemoveContainer" containerID="b56f8f42b658ff23a21eac1f1ea8ae361b5cda69c0f5d4fa4b3be190cfc6d6bb"
Sep 18 20:24:09.816327 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:24:09.816285    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:10.686134 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:24:10.686103    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche72hq6"
Sep 18 20:24:27.918532 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:24:27.918443    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:29.353004 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:24:29.352934    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:31.013836 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:24:31.013798    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:32.320866 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:24:32.320820    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:42.445818 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:24:42.445772    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:45.988743 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:24:45.988699    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche72hq6"]
Sep 18 20:24:45.997210 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:24:45.997175    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:46.095074 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:24:46.095035    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:47.118139 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:24:47.118107    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:47.228883 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:24:47.228844    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:50.685647 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:24:50.685608    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche72hq6" podUID="c7137477-bed9-4efc-bd06-57dcb8e632ce" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:24:56.932224 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:24:56.932178    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:00.685809 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:25:00.685767    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche72hq6" podUID="c7137477-bed9-4efc-bd06-57dcb8e632ce" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:25:01.009228 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:25:01.009132    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche72hq6" podUID="c7137477-bed9-4efc-bd06-57dcb8e632ce" containerName="main" containerID="cri-o://07fec5689a3823fc135a8daa93336e023f353cd20a4f026106f8ba121ac2e17d" gracePeriod=45
Sep 18 20:25:07.537891 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:25:07.537853    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:10.685903 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:25:10.685857    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche72hq6" podUID="c7137477-bed9-4efc-bd06-57dcb8e632ce" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:25:10.686604 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:25:10.686573    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche72hq6"
Sep 18 20:25:12.451998 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:25:12.451956    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:12.602547 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:25:12.602507    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:16.614195 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:25:16.614149    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:16.831190 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:25:16.831154    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:17.500259 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:25:17.500226    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:20.685526 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:25:20.685479    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche72hq6" podUID="c7137477-bed9-4efc-bd06-57dcb8e632ce" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:25:30.686146 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:25:30.686105    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche72hq6" podUID="c7137477-bed9-4efc-bd06-57dcb8e632ce" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:25:31.235682 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:25:31.235643    2564 generic.go:356] "Generic (PLEG): container finished" podID="c7137477-bed9-4efc-bd06-57dcb8e632ce" containerID="07fec5689a3823fc135a8daa93336e023f353cd20a4f026106f8ba121ac2e17d" exitCode=0
Sep 18 20:25:31.235849 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:25:31.235711    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche72hq6" event={"ID":"c7137477-bed9-4efc-bd06-57dcb8e632ce","Type":"ContainerDied","Data":"07fec5689a3823fc135a8daa93336e023f353cd20a4f026106f8ba121ac2e17d"}
Sep 18 20:25:31.265565 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:25:31.265539    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche72hq6"
Sep 18 20:25:31.321267 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:25:31.321168    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7137477-bed9-4efc-bd06-57dcb8e632ce-tls-certs\") pod \"c7137477-bed9-4efc-bd06-57dcb8e632ce\" (UID: \"c7137477-bed9-4efc-bd06-57dcb8e632ce\") "
Sep 18 20:25:31.321429 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:25:31.321307    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r6q58\" (UniqueName: \"kubernetes.io/projected/c7137477-bed9-4efc-bd06-57dcb8e632ce-kube-api-access-r6q58\") pod \"c7137477-bed9-4efc-bd06-57dcb8e632ce\" (UID: \"c7137477-bed9-4efc-bd06-57dcb8e632ce\") "
Sep 18 20:25:31.323517 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:25:31.323476    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7137477-bed9-4efc-bd06-57dcb8e632ce-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c7137477-bed9-4efc-bd06-57dcb8e632ce" (UID: "c7137477-bed9-4efc-bd06-57dcb8e632ce"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:25:31.323645 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:25:31.323592    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7137477-bed9-4efc-bd06-57dcb8e632ce-kube-api-access-r6q58" (OuterVolumeSpecName: "kube-api-access-r6q58") pod "c7137477-bed9-4efc-bd06-57dcb8e632ce" (UID: "c7137477-bed9-4efc-bd06-57dcb8e632ce"). InnerVolumeSpecName "kube-api-access-r6q58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:25:31.422429 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:25:31.422389    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7137477-bed9-4efc-bd06-57dcb8e632ce-tls-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:25:31.422429 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:25:31.422429    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6q58\" (UniqueName: \"kubernetes.io/projected/c7137477-bed9-4efc-bd06-57dcb8e632ce-kube-api-access-r6q58\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:25:32.249607 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:25:32.249565    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche72hq6"
Sep 18 20:25:32.249607 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:25:32.249584    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche72hq6" event={"ID":"c7137477-bed9-4efc-bd06-57dcb8e632ce","Type":"ContainerDied","Data":"cfb50717a6bd97e15da6bfc7e7aa6c4b5fcadb42abf058618e45084e6e0cc454"}
Sep 18 20:25:32.250229 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:25:32.249689    2564 scope.go:117] "RemoveContainer" containerID="07fec5689a3823fc135a8daa93336e023f353cd20a4f026106f8ba121ac2e17d"
Sep 18 20:25:32.266758 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:25:32.266722    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche72hq6"]
Sep 18 20:25:32.273847 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:25:32.273811    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche72hq6"]
Sep 18 20:25:34.001511 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:25:34.001465    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7137477-bed9-4efc-bd06-57dcb8e632ce" path="/var/lib/kubelet/pods/c7137477-bed9-4efc-bd06-57dcb8e632ce/volumes"
Sep 18 20:25:38.683701 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:25:38.683663    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:39.635489 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:25:39.635454    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:40.689017 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:25:40.688952    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:40.828487 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:25:40.828440    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:40.854298 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:25:40.854258    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9gn8q"]
Sep 18 20:25:40.910237 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:25:40.910203    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:44.506494 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:25:44.506456    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9gn8q" podUID="91411429-61a5-42bc-bf80-e87a2b47b68f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:25:46.253171 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:25:46.253134    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:51.034296 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:25:51.034249    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:52.523433 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:25:52.523397    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:52.714525 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:25:52.714478    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:54.506742 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:25:54.506703    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9gn8q" podUID="91411429-61a5-42bc-bf80-e87a2b47b68f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:25:55.874184 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:25:55.874140    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9gn8q" podUID="91411429-61a5-42bc-bf80-e87a2b47b68f" containerName="main" containerID="cri-o://76320f935d5083620377e8e54b3fa5e3db64246ea582fd7f51cf3ec04ef18b91" gracePeriod=45
Sep 18 20:26:04.506407 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:26:04.506368    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9gn8q" podUID="91411429-61a5-42bc-bf80-e87a2b47b68f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:26:04.506964 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:26:04.506475    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9gn8q"
Sep 18 20:26:14.506441 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:26:14.506402    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9gn8q" podUID="91411429-61a5-42bc-bf80-e87a2b47b68f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:26:24.506111 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:26:24.506066    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9gn8q" podUID="91411429-61a5-42bc-bf80-e87a2b47b68f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:26:26.159001 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:26:26.158955    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9gn8q"
Sep 18 20:26:26.311691 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:26:26.311656    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91411429-61a5-42bc-bf80-e87a2b47b68f-tls-certs\") pod \"91411429-61a5-42bc-bf80-e87a2b47b68f\" (UID: \"91411429-61a5-42bc-bf80-e87a2b47b68f\") "
Sep 18 20:26:26.311865 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:26:26.311762    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-27z4h\" (UniqueName: \"kubernetes.io/projected/91411429-61a5-42bc-bf80-e87a2b47b68f-kube-api-access-27z4h\") pod \"91411429-61a5-42bc-bf80-e87a2b47b68f\" (UID: \"91411429-61a5-42bc-bf80-e87a2b47b68f\") "
Sep 18 20:26:26.313870 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:26:26.313837    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91411429-61a5-42bc-bf80-e87a2b47b68f-kube-api-access-27z4h" (OuterVolumeSpecName: "kube-api-access-27z4h") pod "91411429-61a5-42bc-bf80-e87a2b47b68f" (UID: "91411429-61a5-42bc-bf80-e87a2b47b68f"). InnerVolumeSpecName "kube-api-access-27z4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:26:26.313870 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:26:26.313853    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91411429-61a5-42bc-bf80-e87a2b47b68f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "91411429-61a5-42bc-bf80-e87a2b47b68f" (UID: "91411429-61a5-42bc-bf80-e87a2b47b68f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:26:26.413573 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:26:26.413455    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27z4h\" (UniqueName: \"kubernetes.io/projected/91411429-61a5-42bc-bf80-e87a2b47b68f-kube-api-access-27z4h\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:26:26.413573 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:26:26.413508    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91411429-61a5-42bc-bf80-e87a2b47b68f-tls-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:26:26.515711 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:26:26.515671    2564 generic.go:356] "Generic (PLEG): container finished" podID="91411429-61a5-42bc-bf80-e87a2b47b68f" containerID="76320f935d5083620377e8e54b3fa5e3db64246ea582fd7f51cf3ec04ef18b91" exitCode=0
Sep 18 20:26:26.515894 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:26:26.515727    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9gn8q"
Sep 18 20:26:26.515894 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:26:26.515735    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9gn8q" event={"ID":"91411429-61a5-42bc-bf80-e87a2b47b68f","Type":"ContainerDied","Data":"76320f935d5083620377e8e54b3fa5e3db64246ea582fd7f51cf3ec04ef18b91"}
Sep 18 20:26:26.515894 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:26:26.515821    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9gn8q" event={"ID":"91411429-61a5-42bc-bf80-e87a2b47b68f","Type":"ContainerDied","Data":"8dd566af291434b73fcfd9a275d0e74acb9d1c2aafb01249367317e956f5e193"}
Sep 18 20:26:26.515894 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:26:26.515853    2564 scope.go:117] "RemoveContainer" containerID="76320f935d5083620377e8e54b3fa5e3db64246ea582fd7f51cf3ec04ef18b91"
Sep 18 20:26:26.525526 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:26:26.525506    2564 scope.go:117] "RemoveContainer" containerID="76320f935d5083620377e8e54b3fa5e3db64246ea582fd7f51cf3ec04ef18b91"
Sep 18 20:26:26.525837 ip-10-0-128-161 kubenswrapper[2564]: E0918 20:26:26.525812    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76320f935d5083620377e8e54b3fa5e3db64246ea582fd7f51cf3ec04ef18b91\": container with ID starting with 76320f935d5083620377e8e54b3fa5e3db64246ea582fd7f51cf3ec04ef18b91 not found: ID does not exist" containerID="76320f935d5083620377e8e54b3fa5e3db64246ea582fd7f51cf3ec04ef18b91"
Sep 18 20:26:26.525938 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:26:26.525844    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76320f935d5083620377e8e54b3fa5e3db64246ea582fd7f51cf3ec04ef18b91"} err="failed to get container status \"76320f935d5083620377e8e54b3fa5e3db64246ea582fd7f51cf3ec04ef18b91\": rpc error: code = NotFound desc = could not find container \"76320f935d5083620377e8e54b3fa5e3db64246ea582fd7f51cf3ec04ef18b91\": container with ID starting with 76320f935d5083620377e8e54b3fa5e3db64246ea582fd7f51cf3ec04ef18b91 not found: ID does not exist"
Sep 18 20:26:26.544557 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:26:26.544520    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9gn8q"]
Sep 18 20:26:26.549630 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:26:26.549599    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9gn8q"]
Sep 18 20:26:28.001718 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:26:28.001675    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91411429-61a5-42bc-bf80-e87a2b47b68f" path="/var/lib/kubelet/pods/91411429-61a5-42bc-bf80-e87a2b47b68f/volumes"
Sep 18 20:26:31.206796 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:26:31.206761    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:26:31.303864 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:26:31.303825    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:26:36.630533 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:26:36.630487    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:26:36.836882 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:26:36.836848    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schevxkwk"]
Sep 18 20:26:36.837251 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:26:36.837235    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91411429-61a5-42bc-bf80-e87a2b47b68f" containerName="main"
Sep 18 20:26:36.837308 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:26:36.837253    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="91411429-61a5-42bc-bf80-e87a2b47b68f" containerName="main"
Sep 18 20:26:36.837308 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:26:36.837275    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df773247-0709-461f-8d79-866de7a188c5" containerName="main"
Sep 18 20:26:36.837308 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:26:36.837281    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="df773247-0709-461f-8d79-866de7a188c5" containerName="main"
Sep 18 20:26:36.837308 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:26:36.837289    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7137477-bed9-4efc-bd06-57dcb8e632ce" containerName="main"
Sep 18 20:26:36.837308 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:26:36.837294    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7137477-bed9-4efc-bd06-57dcb8e632ce" containerName="main"
Sep 18 20:26:36.837479 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:26:36.837351    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="91411429-61a5-42bc-bf80-e87a2b47b68f" containerName="main"
Sep 18 20:26:36.837479 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:26:36.837363    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="df773247-0709-461f-8d79-866de7a188c5" containerName="main"
Sep 18 20:26:36.837479 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:26:36.837370    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7137477-bed9-4efc-bd06-57dcb8e632ce" containerName="main"
Sep 18 20:26:36.841661 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:26:36.841644    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schevxkwk"
Sep 18 20:26:36.844651 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:26:36.844621    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Sep 18 20:26:36.844651 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:26:36.844636    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Sep 18 20:26:36.844884 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:26:36.844625    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Sep 18 20:26:36.844884 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:26:36.844681    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-sjpw7\""
Sep 18 20:26:36.844884 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:26:36.844716    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa-dockercfg-4k6x6\""
Sep 18 20:26:36.851419 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:26:36.851391    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schevxkwk"]
Sep 18 20:26:36.902047 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:26:36.901933    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:26:37.012478 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:26:37.012440    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4afe1c02-8183-4e7c-8e29-98f1b8cd520e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schevxkwk\" (UID: \"4afe1c02-8183-4e7c-8e29-98f1b8cd520e\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schevxkwk"
Sep 18 20:26:37.012660 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:26:37.012633    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drqdc\" (UniqueName: \"kubernetes.io/projected/4afe1c02-8183-4e7c-8e29-98f1b8cd520e-kube-api-access-drqdc\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schevxkwk\" (UID: \"4afe1c02-8183-4e7c-8e29-98f1b8cd520e\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schevxkwk"
Sep 18 20:26:37.113813 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:26:37.113778    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drqdc\" (UniqueName: \"kubernetes.io/projected/4afe1c02-8183-4e7c-8e29-98f1b8cd520e-kube-api-access-drqdc\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schevxkwk\" (UID: \"4afe1c02-8183-4e7c-8e29-98f1b8cd520e\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schevxkwk"
Sep 18 20:26:37.113813 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:26:37.113826    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4afe1c02-8183-4e7c-8e29-98f1b8cd520e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schevxkwk\" (UID: \"4afe1c02-8183-4e7c-8e29-98f1b8cd520e\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schevxkwk"
Sep 18 20:26:37.116413 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:26:37.116385    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4afe1c02-8183-4e7c-8e29-98f1b8cd520e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schevxkwk\" (UID: \"4afe1c02-8183-4e7c-8e29-98f1b8cd520e\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schevxkwk"
Sep 18 20:26:37.124452 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:26:37.124426    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drqdc\" (UniqueName: \"kubernetes.io/projected/4afe1c02-8183-4e7c-8e29-98f1b8cd520e-kube-api-access-drqdc\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schevxkwk\" (UID: \"4afe1c02-8183-4e7c-8e29-98f1b8cd520e\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schevxkwk"
Sep 18 20:26:37.153396 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:26:37.153316    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schevxkwk"
Sep 18 20:26:37.432017 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:26:37.431820    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schevxkwk"]
Sep 18 20:26:37.434852 ip-10-0-128-161 kubenswrapper[2564]: W0918 20:26:37.434820    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4afe1c02_8183_4e7c_8e29_98f1b8cd520e.slice/crio-37f9a70cfb40365c4ae1d7bfaefdc67ea964d4da520439d7e683d19144396a5f WatchSource:0}: Error finding container 37f9a70cfb40365c4ae1d7bfaefdc67ea964d4da520439d7e683d19144396a5f: Status 404 returned error can't find the container with id 37f9a70cfb40365c4ae1d7bfaefdc67ea964d4da520439d7e683d19144396a5f
Sep 18 20:26:37.578553 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:26:37.578511    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schevxkwk" event={"ID":"4afe1c02-8183-4e7c-8e29-98f1b8cd520e","Type":"ContainerStarted","Data":"c0277f17bfc33a8137f10f24bae5807b2fb5a1bbd39a12b3f163df2f843a361a"}
Sep 18 20:26:37.578722 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:26:37.578585    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schevxkwk" event={"ID":"4afe1c02-8183-4e7c-8e29-98f1b8cd520e","Type":"ContainerStarted","Data":"37f9a70cfb40365c4ae1d7bfaefdc67ea964d4da520439d7e683d19144396a5f"}
Sep 18 20:26:37.578722 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:26:37.578641    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schevxkwk"
Sep 18 20:26:37.607289 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:26:37.607232    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schevxkwk" podStartSLOduration=1.6072052650000002 podStartE2EDuration="1.607205265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:26:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:26:37.605146756 +0000 UTC m=+2914.115502863" watchObservedRunningTime="2026-09-18 20:26:37.607205265 +0000 UTC m=+2914.117561356"
Sep 18 20:26:38.283634 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:26:38.283595    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:26:44.860740 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:26:44.860708    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:26:44.978404 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:26:44.978365    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:26:50.182556 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:26:50.182513    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:27:08.584843 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:27:08.584808    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schevxkwk"
Sep 18 20:27:21.707781 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:27:21.707743    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:27:21.793234 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:27:21.793193    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:28:04.150019 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:28:04.149971    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-58vl4_945a444e-9381-4e2f-b4c1-64be0ec88e6c/console-operator/2.log"
Sep 18 20:28:04.159007 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:28:04.158959    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-58vl4_945a444e-9381-4e2f-b4c1-64be0ec88e6c/console-operator/2.log"
Sep 18 20:28:11.772683 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:28:11.772647    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:28:11.864187 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:28:11.864146    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:28:18.204186 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:28:18.204147    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:28:18.424756 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:28:18.424713    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:28:18.625397 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:28:18.625299    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche24r5g"]
Sep 18 20:28:18.628668 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:28:18.628647    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche24r5g"
Sep 18 20:28:18.632297 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:28:18.632265    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"kube-root-ca.crt\""
Sep 18 20:28:18.633232 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:28:18.633212    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"openshift-service-ca.crt\""
Sep 18 20:28:18.633337 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:28:18.633239    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisv20d1e8ef181cac7effb0fab7b8590b15-kserve-self-signed-certs\""
Sep 18 20:28:18.633337 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:28:18.633220    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisvc-model-fb-opt-125m-oci-r-0fba19de-epp-sa-dockercfg-wfm5x\""
Sep 18 20:28:18.633337 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:28:18.633302    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"default-dockercfg-gchg9\""
Sep 18 20:28:18.642476 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:28:18.642450    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche24r5g"]
Sep 18 20:28:18.664924 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:28:18.664892    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:28:18.719974 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:28:18.719934    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6ff6\" (UniqueName: \"kubernetes.io/projected/d90a0ae8-5fef-4d6c-9ecf-f9d5d6351ef4-kube-api-access-k6ff6\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche24r5g\" (UID: \"d90a0ae8-5fef-4d6c-9ecf-f9d5d6351ef4\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche24r5g"
Sep 18 20:28:18.720189 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:28:18.720148    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d90a0ae8-5fef-4d6c-9ecf-f9d5d6351ef4-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche24r5g\" (UID: \"d90a0ae8-5fef-4d6c-9ecf-f9d5d6351ef4\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche24r5g"
Sep 18 20:28:18.821248 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:28:18.821204    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6ff6\" (UniqueName: \"kubernetes.io/projected/d90a0ae8-5fef-4d6c-9ecf-f9d5d6351ef4-kube-api-access-k6ff6\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche24r5g\" (UID: \"d90a0ae8-5fef-4d6c-9ecf-f9d5d6351ef4\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche24r5g"
Sep 18 20:28:18.821458 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:28:18.821269    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d90a0ae8-5fef-4d6c-9ecf-f9d5d6351ef4-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche24r5g\" (UID: \"d90a0ae8-5fef-4d6c-9ecf-f9d5d6351ef4\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche24r5g"
Sep 18 20:28:18.823869 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:28:18.823839    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d90a0ae8-5fef-4d6c-9ecf-f9d5d6351ef4-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche24r5g\" (UID: \"d90a0ae8-5fef-4d6c-9ecf-f9d5d6351ef4\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche24r5g"
Sep 18 20:28:18.829770 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:28:18.829733    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6ff6\" (UniqueName: \"kubernetes.io/projected/d90a0ae8-5fef-4d6c-9ecf-f9d5d6351ef4-kube-api-access-k6ff6\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche24r5g\" (UID: \"d90a0ae8-5fef-4d6c-9ecf-f9d5d6351ef4\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche24r5g"
Sep 18 20:28:18.939095 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:28:18.939059    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche24r5g"
Sep 18 20:28:19.227767 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:28:19.227734    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche24r5g"]
Sep 18 20:28:19.228142 ip-10-0-128-161 kubenswrapper[2564]: W0918 20:28:19.228112    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd90a0ae8_5fef_4d6c_9ecf_f9d5d6351ef4.slice/crio-f0c52186b32a09ce6ad31c4d43ab438854d5a6ddbca99ad47046c6cbcfbac963 WatchSource:0}: Error finding container f0c52186b32a09ce6ad31c4d43ab438854d5a6ddbca99ad47046c6cbcfbac963: Status 404 returned error can't find the container with id f0c52186b32a09ce6ad31c4d43ab438854d5a6ddbca99ad47046c6cbcfbac963
Sep 18 20:28:20.090022 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:28:20.089959    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche24r5g" event={"ID":"d90a0ae8-5fef-4d6c-9ecf-f9d5d6351ef4","Type":"ContainerStarted","Data":"afa3137fc1a7533af73cc22c8ea19095192b2bb380d77eff65f83ce8fbc7b608"}
Sep 18 20:28:20.090239 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:28:20.090035    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche24r5g" event={"ID":"d90a0ae8-5fef-4d6c-9ecf-f9d5d6351ef4","Type":"ContainerStarted","Data":"f0c52186b32a09ce6ad31c4d43ab438854d5a6ddbca99ad47046c6cbcfbac963"}
Sep 18 20:28:20.090239 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:28:20.090066    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche24r5g"
Sep 18 20:28:20.111807 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:28:20.111740    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche24r5g" podStartSLOduration=2.111717485 podStartE2EDuration="2.111717485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:28:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:28:20.109106539 +0000 UTC m=+3016.619462651" watchObservedRunningTime="2026-09-18 20:28:20.111717485 +0000 UTC m=+3016.622073577"
Sep 18 20:28:22.596876 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:28:22.596822    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:28:22.672782 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:28:22.672744    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:28:24.191685 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:28:24.191643    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:28:24.290110 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:28:24.290068    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:28:49.633192 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:28:49.633152    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:28:51.098498 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:28:51.098468    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche24r5g"
Sep 18 20:28:51.440501 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:28:51.440463    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:28:51.441807 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:28:51.441780    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schevxkwk"]
Sep 18 20:28:51.502871 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:28:51.502834    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:28:58.585064 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:28:58.584961    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schevxkwk" podUID="4afe1c02-8183-4e7c-8e29-98f1b8cd520e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:29:06.462339 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:29:06.462291    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schevxkwk" podUID="4afe1c02-8183-4e7c-8e29-98f1b8cd520e" containerName="main" containerID="cri-o://c0277f17bfc33a8137f10f24bae5807b2fb5a1bbd39a12b3f163df2f843a361a" gracePeriod=45
Sep 18 20:29:08.585457 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:29:08.585416    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schevxkwk" podUID="4afe1c02-8183-4e7c-8e29-98f1b8cd520e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:29:18.585244 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:29:18.585198    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schevxkwk" podUID="4afe1c02-8183-4e7c-8e29-98f1b8cd520e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:29:18.585648 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:29:18.585303    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schevxkwk"
Sep 18 20:29:28.584933 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:29:28.584888    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schevxkwk" podUID="4afe1c02-8183-4e7c-8e29-98f1b8cd520e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:29:36.725168 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:29:36.725096    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schevxkwk"
Sep 18 20:29:36.835699 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:29:36.835660    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-drqdc\" (UniqueName: \"kubernetes.io/projected/4afe1c02-8183-4e7c-8e29-98f1b8cd520e-kube-api-access-drqdc\") pod \"4afe1c02-8183-4e7c-8e29-98f1b8cd520e\" (UID: \"4afe1c02-8183-4e7c-8e29-98f1b8cd520e\") "
Sep 18 20:29:36.835914 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:29:36.835738    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4afe1c02-8183-4e7c-8e29-98f1b8cd520e-tls-certs\") pod \"4afe1c02-8183-4e7c-8e29-98f1b8cd520e\" (UID: \"4afe1c02-8183-4e7c-8e29-98f1b8cd520e\") "
Sep 18 20:29:36.838189 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:29:36.838150    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4afe1c02-8183-4e7c-8e29-98f1b8cd520e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4afe1c02-8183-4e7c-8e29-98f1b8cd520e" (UID: "4afe1c02-8183-4e7c-8e29-98f1b8cd520e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:29:36.838334 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:29:36.838198    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4afe1c02-8183-4e7c-8e29-98f1b8cd520e-kube-api-access-drqdc" (OuterVolumeSpecName: "kube-api-access-drqdc") pod "4afe1c02-8183-4e7c-8e29-98f1b8cd520e" (UID: "4afe1c02-8183-4e7c-8e29-98f1b8cd520e"). InnerVolumeSpecName "kube-api-access-drqdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:29:36.936976 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:29:36.936920    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4afe1c02-8183-4e7c-8e29-98f1b8cd520e-tls-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:29:36.936976 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:29:36.936969    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drqdc\" (UniqueName: \"kubernetes.io/projected/4afe1c02-8183-4e7c-8e29-98f1b8cd520e-kube-api-access-drqdc\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:29:37.473370 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:29:37.473326    2564 generic.go:356] "Generic (PLEG): container finished" podID="4afe1c02-8183-4e7c-8e29-98f1b8cd520e" containerID="c0277f17bfc33a8137f10f24bae5807b2fb5a1bbd39a12b3f163df2f843a361a" exitCode=0
Sep 18 20:29:37.473557 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:29:37.473378    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schevxkwk"
Sep 18 20:29:37.473557 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:29:37.473412    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schevxkwk" event={"ID":"4afe1c02-8183-4e7c-8e29-98f1b8cd520e","Type":"ContainerDied","Data":"c0277f17bfc33a8137f10f24bae5807b2fb5a1bbd39a12b3f163df2f843a361a"}
Sep 18 20:29:37.473557 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:29:37.473464    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schevxkwk" event={"ID":"4afe1c02-8183-4e7c-8e29-98f1b8cd520e","Type":"ContainerDied","Data":"37f9a70cfb40365c4ae1d7bfaefdc67ea964d4da520439d7e683d19144396a5f"}
Sep 18 20:29:37.473557 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:29:37.473487    2564 scope.go:117] "RemoveContainer" containerID="c0277f17bfc33a8137f10f24bae5807b2fb5a1bbd39a12b3f163df2f843a361a"
Sep 18 20:29:37.483092 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:29:37.483075    2564 scope.go:117] "RemoveContainer" containerID="c0277f17bfc33a8137f10f24bae5807b2fb5a1bbd39a12b3f163df2f843a361a"
Sep 18 20:29:37.483377 ip-10-0-128-161 kubenswrapper[2564]: E0918 20:29:37.483361    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0277f17bfc33a8137f10f24bae5807b2fb5a1bbd39a12b3f163df2f843a361a\": container with ID starting with c0277f17bfc33a8137f10f24bae5807b2fb5a1bbd39a12b3f163df2f843a361a not found: ID does not exist" containerID="c0277f17bfc33a8137f10f24bae5807b2fb5a1bbd39a12b3f163df2f843a361a"
Sep 18 20:29:37.483420 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:29:37.483387    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0277f17bfc33a8137f10f24bae5807b2fb5a1bbd39a12b3f163df2f843a361a"} err="failed to get container status \"c0277f17bfc33a8137f10f24bae5807b2fb5a1bbd39a12b3f163df2f843a361a\": rpc error: code = NotFound desc = could not find container \"c0277f17bfc33a8137f10f24bae5807b2fb5a1bbd39a12b3f163df2f843a361a\": container with ID starting with c0277f17bfc33a8137f10f24bae5807b2fb5a1bbd39a12b3f163df2f843a361a not found: ID does not exist"
Sep 18 20:29:37.496821 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:29:37.496793    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schevxkwk"]
Sep 18 20:29:37.502911 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:29:37.502886    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schevxkwk"]
Sep 18 20:29:38.000204 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:29:38.000171    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4afe1c02-8183-4e7c-8e29-98f1b8cd520e" path="/var/lib/kubelet/pods/4afe1c02-8183-4e7c-8e29-98f1b8cd520e/volumes"
Sep 18 20:29:41.416135 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:29:41.416084    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:29:41.509475 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:29:41.509432    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:29:46.600185 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:29:46.600122    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:29:46.764086 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:29:46.764047    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche8nfw2"]
Sep 18 20:29:46.764950 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:29:46.764918    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4afe1c02-8183-4e7c-8e29-98f1b8cd520e" containerName="main"
Sep 18 20:29:46.764950 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:29:46.764950    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4afe1c02-8183-4e7c-8e29-98f1b8cd520e" containerName="main"
Sep 18 20:29:46.765155 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:29:46.765103    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4afe1c02-8183-4e7c-8e29-98f1b8cd520e" containerName="main"
Sep 18 20:29:46.768718 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:29:46.768691    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche8nfw2"
Sep 18 20:29:46.771441 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:29:46.771409    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Sep 18 20:29:46.771586 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:29:46.771425    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Sep 18 20:29:46.772217 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:29:46.772162    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-9pdbw\""
Sep 18 20:29:46.772217 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:29:46.772213    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa-dockercfg-jgb4n\""
Sep 18 20:29:46.772398 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:29:46.772257    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Sep 18 20:29:46.779173 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:29:46.779146    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche8nfw2"]
Sep 18 20:29:46.794807 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:29:46.794774    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:29:46.923774 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:29:46.923727    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdpl2\" (UniqueName: \"kubernetes.io/projected/819565a9-7287-4455-8882-bdce4c8912cd-kube-api-access-rdpl2\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche8nfw2\" (UID: \"819565a9-7287-4455-8882-bdce4c8912cd\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche8nfw2"
Sep 18 20:29:46.923964 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:29:46.923806    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/819565a9-7287-4455-8882-bdce4c8912cd-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche8nfw2\" (UID: \"819565a9-7287-4455-8882-bdce4c8912cd\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche8nfw2"
Sep 18 20:29:47.024770 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:29:47.024727    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdpl2\" (UniqueName: \"kubernetes.io/projected/819565a9-7287-4455-8882-bdce4c8912cd-kube-api-access-rdpl2\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche8nfw2\" (UID: \"819565a9-7287-4455-8882-bdce4c8912cd\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche8nfw2"
Sep 18 20:29:47.024955 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:29:47.024789    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/819565a9-7287-4455-8882-bdce4c8912cd-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche8nfw2\" (UID: \"819565a9-7287-4455-8882-bdce4c8912cd\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche8nfw2"
Sep 18 20:29:47.027438 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:29:47.027415    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/819565a9-7287-4455-8882-bdce4c8912cd-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche8nfw2\" (UID: \"819565a9-7287-4455-8882-bdce4c8912cd\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche8nfw2"
Sep 18 20:29:47.033633 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:29:47.033595    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdpl2\" (UniqueName: \"kubernetes.io/projected/819565a9-7287-4455-8882-bdce4c8912cd-kube-api-access-rdpl2\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche8nfw2\" (UID: \"819565a9-7287-4455-8882-bdce4c8912cd\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche8nfw2"
Sep 18 20:29:47.082248 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:29:47.082206    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche8nfw2"
Sep 18 20:29:47.356126 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:29:47.356049    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche8nfw2"]
Sep 18 20:29:47.359249 ip-10-0-128-161 kubenswrapper[2564]: W0918 20:29:47.359210    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod819565a9_7287_4455_8882_bdce4c8912cd.slice/crio-fcdd1abf779bf27d650e9ab9d55423ff98254b436d9000f814cb8c84a49666f4 WatchSource:0}: Error finding container fcdd1abf779bf27d650e9ab9d55423ff98254b436d9000f814cb8c84a49666f4: Status 404 returned error can't find the container with id fcdd1abf779bf27d650e9ab9d55423ff98254b436d9000f814cb8c84a49666f4
Sep 18 20:29:47.528944 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:29:47.528907    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche8nfw2" event={"ID":"819565a9-7287-4455-8882-bdce4c8912cd","Type":"ContainerStarted","Data":"1de6c79b19820144589ad5e1e0fc898b9c24c89ba11e85a110e16a7ca24bf607"}
Sep 18 20:29:47.529164 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:29:47.528971    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche8nfw2" event={"ID":"819565a9-7287-4455-8882-bdce4c8912cd","Type":"ContainerStarted","Data":"fcdd1abf779bf27d650e9ab9d55423ff98254b436d9000f814cb8c84a49666f4"}
Sep 18 20:29:47.529164 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:29:47.529030    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche8nfw2"
Sep 18 20:29:47.551877 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:29:47.551810    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche8nfw2" podStartSLOduration=1.551786791 podStartE2EDuration="1.551786791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:29:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:29:47.550731308 +0000 UTC m=+3104.061087426" watchObservedRunningTime="2026-09-18 20:29:47.551786791 +0000 UTC m=+3104.062142879"
Sep 18 20:29:48.533324 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:29:48.533283    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:29:54.060595 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:29:54.060532    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:29:54.127827 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:29:54.127790    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:29:58.239143 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:29:58.239101    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:30:18.536320 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:30:18.536286    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche8nfw2"
Sep 18 20:30:23.972380 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:30:23.972343    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche24r5g"]
Sep 18 20:30:23.974186 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:30:23.974155    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:30:24.035994 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:30:24.035946    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:30:31.098563 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:30:31.098463    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche24r5g" podUID="d90a0ae8-5fef-4d6c-9ecf-f9d5d6351ef4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:30:38.992919 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:30:38.992877    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche24r5g" podUID="d90a0ae8-5fef-4d6c-9ecf-f9d5d6351ef4" containerName="main" containerID="cri-o://afa3137fc1a7533af73cc22c8ea19095192b2bb380d77eff65f83ce8fbc7b608" gracePeriod=45
Sep 18 20:30:41.098659 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:30:41.098620    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche24r5g" podUID="d90a0ae8-5fef-4d6c-9ecf-f9d5d6351ef4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:30:51.098427 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:30:51.098390    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche24r5g" podUID="d90a0ae8-5fef-4d6c-9ecf-f9d5d6351ef4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:30:51.100537 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:30:51.098489    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche24r5g"
Sep 18 20:31:01.098153 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:31:01.098111    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche24r5g" podUID="d90a0ae8-5fef-4d6c-9ecf-f9d5d6351ef4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:31:07.892805 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:31:07.892751    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:31:08.656001 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:31:08.655940    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche8nfw2"]
Sep 18 20:31:09.074408 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:31:09.074315    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:31:09.259568 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:31:09.259496    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche24r5g"
Sep 18 20:31:09.354996 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:31:09.354957    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d90a0ae8-5fef-4d6c-9ecf-f9d5d6351ef4-tls-certs\") pod \"d90a0ae8-5fef-4d6c-9ecf-f9d5d6351ef4\" (UID: \"d90a0ae8-5fef-4d6c-9ecf-f9d5d6351ef4\") "
Sep 18 20:31:09.355180 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:31:09.355047    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k6ff6\" (UniqueName: \"kubernetes.io/projected/d90a0ae8-5fef-4d6c-9ecf-f9d5d6351ef4-kube-api-access-k6ff6\") pod \"d90a0ae8-5fef-4d6c-9ecf-f9d5d6351ef4\" (UID: \"d90a0ae8-5fef-4d6c-9ecf-f9d5d6351ef4\") "
Sep 18 20:31:09.357362 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:31:09.357314    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d90a0ae8-5fef-4d6c-9ecf-f9d5d6351ef4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d90a0ae8-5fef-4d6c-9ecf-f9d5d6351ef4" (UID: "d90a0ae8-5fef-4d6c-9ecf-f9d5d6351ef4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:31:09.357499 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:31:09.357357    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d90a0ae8-5fef-4d6c-9ecf-f9d5d6351ef4-kube-api-access-k6ff6" (OuterVolumeSpecName: "kube-api-access-k6ff6") pod "d90a0ae8-5fef-4d6c-9ecf-f9d5d6351ef4" (UID: "d90a0ae8-5fef-4d6c-9ecf-f9d5d6351ef4"). InnerVolumeSpecName "kube-api-access-k6ff6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:31:09.455683 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:31:09.455632    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d90a0ae8-5fef-4d6c-9ecf-f9d5d6351ef4-tls-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:31:09.455683 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:31:09.455674    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6ff6\" (UniqueName: \"kubernetes.io/projected/d90a0ae8-5fef-4d6c-9ecf-f9d5d6351ef4-kube-api-access-k6ff6\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:31:09.926587 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:31:09.926545    2564 generic.go:356] "Generic (PLEG): container finished" podID="d90a0ae8-5fef-4d6c-9ecf-f9d5d6351ef4" containerID="afa3137fc1a7533af73cc22c8ea19095192b2bb380d77eff65f83ce8fbc7b608" exitCode=0
Sep 18 20:31:09.926774 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:31:09.926603    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche24r5g"
Sep 18 20:31:09.926774 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:31:09.926633    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche24r5g" event={"ID":"d90a0ae8-5fef-4d6c-9ecf-f9d5d6351ef4","Type":"ContainerDied","Data":"afa3137fc1a7533af73cc22c8ea19095192b2bb380d77eff65f83ce8fbc7b608"}
Sep 18 20:31:09.926774 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:31:09.926719    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche24r5g" event={"ID":"d90a0ae8-5fef-4d6c-9ecf-f9d5d6351ef4","Type":"ContainerDied","Data":"f0c52186b32a09ce6ad31c4d43ab438854d5a6ddbca99ad47046c6cbcfbac963"}
Sep 18 20:31:09.926774 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:31:09.926743    2564 scope.go:117] "RemoveContainer" containerID="afa3137fc1a7533af73cc22c8ea19095192b2bb380d77eff65f83ce8fbc7b608"
Sep 18 20:31:09.937015 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:31:09.936963    2564 scope.go:117] "RemoveContainer" containerID="afa3137fc1a7533af73cc22c8ea19095192b2bb380d77eff65f83ce8fbc7b608"
Sep 18 20:31:09.937395 ip-10-0-128-161 kubenswrapper[2564]: E0918 20:31:09.937373    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afa3137fc1a7533af73cc22c8ea19095192b2bb380d77eff65f83ce8fbc7b608\": container with ID starting with afa3137fc1a7533af73cc22c8ea19095192b2bb380d77eff65f83ce8fbc7b608 not found: ID does not exist" containerID="afa3137fc1a7533af73cc22c8ea19095192b2bb380d77eff65f83ce8fbc7b608"
Sep 18 20:31:09.937468 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:31:09.937406    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afa3137fc1a7533af73cc22c8ea19095192b2bb380d77eff65f83ce8fbc7b608"} err="failed to get container status \"afa3137fc1a7533af73cc22c8ea19095192b2bb380d77eff65f83ce8fbc7b608\": rpc error: code = NotFound desc = could not find container \"afa3137fc1a7533af73cc22c8ea19095192b2bb380d77eff65f83ce8fbc7b608\": container with ID starting with afa3137fc1a7533af73cc22c8ea19095192b2bb380d77eff65f83ce8fbc7b608 not found: ID does not exist"
Sep 18 20:31:09.948716 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:31:09.948680    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche24r5g"]
Sep 18 20:31:09.952636 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:31:09.952603    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche24r5g"]
Sep 18 20:31:10.000374 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:31:10.000339    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d90a0ae8-5fef-4d6c-9ecf-f9d5d6351ef4" path="/var/lib/kubelet/pods/d90a0ae8-5fef-4d6c-9ecf-f9d5d6351ef4/volumes"
Sep 18 20:31:14.113952 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:31:14.113913    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:31:14.271932 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:31:14.271894    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:31:18.536777 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:31:18.536733    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche8nfw2" podUID="819565a9-7287-4455-8882-bdce4c8912cd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:31:20.259836 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:31:20.259800    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:31:23.676183 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:31:23.676133    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche8nfw2" podUID="819565a9-7287-4455-8882-bdce4c8912cd" containerName="main" containerID="cri-o://1de6c79b19820144589ad5e1e0fc898b9c24c89ba11e85a110e16a7ca24bf607" gracePeriod=45
Sep 18 20:31:26.317078 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:31:26.317028    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:31:26.422644 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:31:26.422609    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:31:27.464043 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:31:27.464004    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:31:27.748262 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:31:27.748164    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:31:28.536894 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:31:28.536845    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche8nfw2" podUID="819565a9-7287-4455-8882-bdce4c8912cd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:31:38.536623 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:31:38.536582    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche8nfw2" podUID="819565a9-7287-4455-8882-bdce4c8912cd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:31:38.537067 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:31:38.536691    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche8nfw2"
Sep 18 20:31:39.132337 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:31:39.132302    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:31:48.536889 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:31:48.536843    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche8nfw2" podUID="819565a9-7287-4455-8882-bdce4c8912cd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:31:52.351482 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:31:52.351436    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:31:53.937542 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:31:53.937474    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche8nfw2"
Sep 18 20:31:54.060012 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:31:54.059944    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/819565a9-7287-4455-8882-bdce4c8912cd-tls-certs\") pod \"819565a9-7287-4455-8882-bdce4c8912cd\" (UID: \"819565a9-7287-4455-8882-bdce4c8912cd\") "
Sep 18 20:31:54.060216 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:31:54.060026    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rdpl2\" (UniqueName: \"kubernetes.io/projected/819565a9-7287-4455-8882-bdce4c8912cd-kube-api-access-rdpl2\") pod \"819565a9-7287-4455-8882-bdce4c8912cd\" (UID: \"819565a9-7287-4455-8882-bdce4c8912cd\") "
Sep 18 20:31:54.062366 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:31:54.062330    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/819565a9-7287-4455-8882-bdce4c8912cd-kube-api-access-rdpl2" (OuterVolumeSpecName: "kube-api-access-rdpl2") pod "819565a9-7287-4455-8882-bdce4c8912cd" (UID: "819565a9-7287-4455-8882-bdce4c8912cd"). InnerVolumeSpecName "kube-api-access-rdpl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:31:54.062366 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:31:54.062354    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/819565a9-7287-4455-8882-bdce4c8912cd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "819565a9-7287-4455-8882-bdce4c8912cd" (UID: "819565a9-7287-4455-8882-bdce4c8912cd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:31:54.157336 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:31:54.157300    2564 generic.go:356] "Generic (PLEG): container finished" podID="819565a9-7287-4455-8882-bdce4c8912cd" containerID="1de6c79b19820144589ad5e1e0fc898b9c24c89ba11e85a110e16a7ca24bf607" exitCode=0
Sep 18 20:31:54.157523 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:31:54.157356    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche8nfw2"
Sep 18 20:31:54.157523 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:31:54.157377    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche8nfw2" event={"ID":"819565a9-7287-4455-8882-bdce4c8912cd","Type":"ContainerDied","Data":"1de6c79b19820144589ad5e1e0fc898b9c24c89ba11e85a110e16a7ca24bf607"}
Sep 18 20:31:54.157523 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:31:54.157430    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche8nfw2" event={"ID":"819565a9-7287-4455-8882-bdce4c8912cd","Type":"ContainerDied","Data":"fcdd1abf779bf27d650e9ab9d55423ff98254b436d9000f814cb8c84a49666f4"}
Sep 18 20:31:54.157523 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:31:54.157453    2564 scope.go:117] "RemoveContainer" containerID="1de6c79b19820144589ad5e1e0fc898b9c24c89ba11e85a110e16a7ca24bf607"
Sep 18 20:31:54.160777 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:31:54.160747    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdpl2\" (UniqueName: \"kubernetes.io/projected/819565a9-7287-4455-8882-bdce4c8912cd-kube-api-access-rdpl2\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:31:54.160777 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:31:54.160769    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/819565a9-7287-4455-8882-bdce4c8912cd-tls-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:31:54.167174 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:31:54.167154    2564 scope.go:117] "RemoveContainer" containerID="1de6c79b19820144589ad5e1e0fc898b9c24c89ba11e85a110e16a7ca24bf607"
Sep 18 20:31:54.167506 ip-10-0-128-161 kubenswrapper[2564]: E0918 20:31:54.167488    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1de6c79b19820144589ad5e1e0fc898b9c24c89ba11e85a110e16a7ca24bf607\": container with ID starting with 1de6c79b19820144589ad5e1e0fc898b9c24c89ba11e85a110e16a7ca24bf607 not found: ID does not exist" containerID="1de6c79b19820144589ad5e1e0fc898b9c24c89ba11e85a110e16a7ca24bf607"
Sep 18 20:31:54.167566 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:31:54.167517    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1de6c79b19820144589ad5e1e0fc898b9c24c89ba11e85a110e16a7ca24bf607"} err="failed to get container status \"1de6c79b19820144589ad5e1e0fc898b9c24c89ba11e85a110e16a7ca24bf607\": rpc error: code = NotFound desc = could not find container \"1de6c79b19820144589ad5e1e0fc898b9c24c89ba11e85a110e16a7ca24bf607\": container with ID starting with 1de6c79b19820144589ad5e1e0fc898b9c24c89ba11e85a110e16a7ca24bf607 not found: ID does not exist"
Sep 18 20:31:54.179487 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:31:54.179449    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche8nfw2"]
Sep 18 20:31:54.183426 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:31:54.183397    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche8nfw2"]
Sep 18 20:31:55.999768 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:31:55.999674    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="819565a9-7287-4455-8882-bdce4c8912cd" path="/var/lib/kubelet/pods/819565a9-7287-4455-8882-bdce4c8912cd/volumes"
Sep 18 20:31:58.737764 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:31:58.737717    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:31:58.840995 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:31:58.840923    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:31:58.938793 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:31:58.938751    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:32:01.021640 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:01.021601    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:32:01.160466 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:01.160424    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:32:01.291207 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:01.291115    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:32:04.758251 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:04.758213    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:32:05.934462 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:05.934428    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:32:06.128592 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:06.128552    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-8n4dh"]
Sep 18 20:32:06.129100 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:06.129076    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d90a0ae8-5fef-4d6c-9ecf-f9d5d6351ef4" containerName="main"
Sep 18 20:32:06.129254 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:06.129175    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d90a0ae8-5fef-4d6c-9ecf-f9d5d6351ef4" containerName="main"
Sep 18 20:32:06.129254 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:06.129198    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="819565a9-7287-4455-8882-bdce4c8912cd" containerName="main"
Sep 18 20:32:06.129254 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:06.129206    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="819565a9-7287-4455-8882-bdce4c8912cd" containerName="main"
Sep 18 20:32:06.129431 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:06.129360    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d90a0ae8-5fef-4d6c-9ecf-f9d5d6351ef4" containerName="main"
Sep 18 20:32:06.129431 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:06.129377    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="819565a9-7287-4455-8882-bdce4c8912cd" containerName="main"
Sep 18 20:32:06.133868 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:06.133837    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-8n4dh"
Sep 18 20:32:06.136907 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:06.136871    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"openshift-service-ca.crt\""
Sep 18 20:32:06.136907 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:06.136890    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"e2e-del-blocked-svc-epp-sa-dockercfg-sd5zs\""
Sep 18 20:32:06.137166 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:06.136921    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"default-dockercfg-d7xmz\""
Sep 18 20:32:06.137166 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:06.136924    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"e2e-del-blocked-svc-kserve-self-signed-certs\""
Sep 18 20:32:06.137307 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:06.137283    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"kube-root-ca.crt\""
Sep 18 20:32:06.145194 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:06.145158    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-8n4dh"]
Sep 18 20:32:06.173389 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:06.173348    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:32:06.266435 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:06.266334    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzn2s\" (UniqueName: \"kubernetes.io/projected/6fb50972-c126-4a2c-9b1b-dedf882cd1b0-kube-api-access-xzn2s\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-8n4dh\" (UID: \"6fb50972-c126-4a2c-9b1b-dedf882cd1b0\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-8n4dh"
Sep 18 20:32:06.266435 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:06.266384    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fb50972-c126-4a2c-9b1b-dedf882cd1b0-tls-certs\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-8n4dh\" (UID: \"6fb50972-c126-4a2c-9b1b-dedf882cd1b0\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-8n4dh"
Sep 18 20:32:06.367477 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:06.367432    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzn2s\" (UniqueName: \"kubernetes.io/projected/6fb50972-c126-4a2c-9b1b-dedf882cd1b0-kube-api-access-xzn2s\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-8n4dh\" (UID: \"6fb50972-c126-4a2c-9b1b-dedf882cd1b0\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-8n4dh"
Sep 18 20:32:06.367477 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:06.367481    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fb50972-c126-4a2c-9b1b-dedf882cd1b0-tls-certs\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-8n4dh\" (UID: \"6fb50972-c126-4a2c-9b1b-dedf882cd1b0\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-8n4dh"
Sep 18 20:32:06.370293 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:06.370264    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fb50972-c126-4a2c-9b1b-dedf882cd1b0-tls-certs\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-8n4dh\" (UID: \"6fb50972-c126-4a2c-9b1b-dedf882cd1b0\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-8n4dh"
Sep 18 20:32:06.381087 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:06.381051    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzn2s\" (UniqueName: \"kubernetes.io/projected/6fb50972-c126-4a2c-9b1b-dedf882cd1b0-kube-api-access-xzn2s\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-8n4dh\" (UID: \"6fb50972-c126-4a2c-9b1b-dedf882cd1b0\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-8n4dh"
Sep 18 20:32:06.447771 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:06.447719    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-8n4dh"
Sep 18 20:32:06.722883 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:06.722848    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-8n4dh"]
Sep 18 20:32:06.725817 ip-10-0-128-161 kubenswrapper[2564]: W0918 20:32:06.725780    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fb50972_c126_4a2c_9b1b_dedf882cd1b0.slice/crio-070612c64b2c1325a86c71db19a0d76f790a819b1500faea0f842543f7a829d2 WatchSource:0}: Error finding container 070612c64b2c1325a86c71db19a0d76f790a819b1500faea0f842543f7a829d2: Status 404 returned error can't find the container with id 070612c64b2c1325a86c71db19a0d76f790a819b1500faea0f842543f7a829d2
Sep 18 20:32:07.222610 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:07.222567    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-8n4dh" event={"ID":"6fb50972-c126-4a2c-9b1b-dedf882cd1b0","Type":"ContainerStarted","Data":"fe0f0337106624b000fae30b51465fd2129f5f457ebfd6e3a384063426a60ae4"}
Sep 18 20:32:07.223056 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:07.222622    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-8n4dh" event={"ID":"6fb50972-c126-4a2c-9b1b-dedf882cd1b0","Type":"ContainerStarted","Data":"070612c64b2c1325a86c71db19a0d76f790a819b1500faea0f842543f7a829d2"}
Sep 18 20:32:07.223056 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:07.222715    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-8n4dh"
Sep 18 20:32:07.260072 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:07.260003    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-8n4dh" podStartSLOduration=1.259963987 podStartE2EDuration="1.259963987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:32:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:32:07.259801604 +0000 UTC m=+3243.770157721" watchObservedRunningTime="2026-09-18 20:32:07.259963987 +0000 UTC m=+3243.770320079"
Sep 18 20:32:08.176466 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:08.176431    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:32:08.280259 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:08.280223    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:32:09.018609 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:09.018568    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-8n4dh"]
Sep 18 20:32:09.019784 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:09.019758    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:32:09.165762 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:09.165711    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:32:09.741473 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:09.741438    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:32:09.837394 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:09.837335    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:32:09.934379 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:09.934335    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:32:10.726849 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:10.726809    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:32:10.786473 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:10.786435    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:32:13.963947 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:13.963888    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:32:14.066387 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:14.066348    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:32:14.166116 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:14.166077    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:32:14.811890 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:14.811850    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:32:18.310428 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:18.310395    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:32:18.490004 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:18.489954    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:32:18.591531 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:18.591409    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:32:19.968767 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:19.968724    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:32:20.045721 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:20.045670    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:32:23.925907 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:23.925871    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:32:24.026421 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:24.026372    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:32:24.253039 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:24.252933    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-8n4dh" podUID="6fb50972-c126-4a2c-9b1b-dedf882cd1b0" containerName="main" containerID="cri-o://fe0f0337106624b000fae30b51465fd2129f5f457ebfd6e3a384063426a60ae4" gracePeriod=45
Sep 18 20:32:39.237324 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:39.237284    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-8n4dh" podUID="6fb50972-c126-4a2c-9b1b-dedf882cd1b0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:32:47.427530 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:47.427483    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:32:47.612146 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:47.612105    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddxxdqf"]
Sep 18 20:32:47.615479 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:47.615451    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddxxdqf"
Sep 18 20:32:47.619088 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:47.619055    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"openshift-service-ca.crt\""
Sep 18 20:32:47.619088 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:47.619073    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"e2e-wk-cfg-implicit-svc-kserve-self-signed-certs\""
Sep 18 20:32:47.619320 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:47.619102    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"e2e-wk-cfg-implicit-svc-epp-sa-dockercfg-2j495\""
Sep 18 20:32:47.619320 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:47.619215    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"default-dockercfg-pwd5r\""
Sep 18 20:32:47.619320 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:47.619271    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"kube-root-ca.crt\""
Sep 18 20:32:47.628336 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:47.628301    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddxxdqf"]
Sep 18 20:32:47.643148 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:47.643111    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:32:47.735865 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:47.735771    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5v9b\" (UniqueName: \"kubernetes.io/projected/ef15392d-8249-4cfc-a52b-7465b8219c96-kube-api-access-c5v9b\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddxxdqf\" (UID: \"ef15392d-8249-4cfc-a52b-7465b8219c96\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddxxdqf"
Sep 18 20:32:47.735865 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:47.735821    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef15392d-8249-4cfc-a52b-7465b8219c96-tls-certs\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddxxdqf\" (UID: \"ef15392d-8249-4cfc-a52b-7465b8219c96\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddxxdqf"
Sep 18 20:32:47.836716 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:47.836668    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5v9b\" (UniqueName: \"kubernetes.io/projected/ef15392d-8249-4cfc-a52b-7465b8219c96-kube-api-access-c5v9b\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddxxdqf\" (UID: \"ef15392d-8249-4cfc-a52b-7465b8219c96\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddxxdqf"
Sep 18 20:32:47.836716 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:47.836722    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef15392d-8249-4cfc-a52b-7465b8219c96-tls-certs\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddxxdqf\" (UID: \"ef15392d-8249-4cfc-a52b-7465b8219c96\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddxxdqf"
Sep 18 20:32:47.839352 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:47.839318    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef15392d-8249-4cfc-a52b-7465b8219c96-tls-certs\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddxxdqf\" (UID: \"ef15392d-8249-4cfc-a52b-7465b8219c96\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddxxdqf"
Sep 18 20:32:47.844893 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:47.844866    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5v9b\" (UniqueName: \"kubernetes.io/projected/ef15392d-8249-4cfc-a52b-7465b8219c96-kube-api-access-c5v9b\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddxxdqf\" (UID: \"ef15392d-8249-4cfc-a52b-7465b8219c96\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddxxdqf"
Sep 18 20:32:47.928402 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:47.928365    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddxxdqf"
Sep 18 20:32:48.202597 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:48.202552    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddxxdqf"]
Sep 18 20:32:48.205613 ip-10-0-128-161 kubenswrapper[2564]: W0918 20:32:48.205586    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef15392d_8249_4cfc_a52b_7465b8219c96.slice/crio-233eaf4c4c508d321525cb45d35c2273ae5700544eedfb60a4ef1cafc6aaf8fe WatchSource:0}: Error finding container 233eaf4c4c508d321525cb45d35c2273ae5700544eedfb60a4ef1cafc6aaf8fe: Status 404 returned error can't find the container with id 233eaf4c4c508d321525cb45d35c2273ae5700544eedfb60a4ef1cafc6aaf8fe
Sep 18 20:32:48.425351 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:48.425307    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddxxdqf" event={"ID":"ef15392d-8249-4cfc-a52b-7465b8219c96","Type":"ContainerStarted","Data":"7a16db9be3f6fe2f30e140d39ce1a5b786e18dc733163ceac543e781441a5759"}
Sep 18 20:32:48.425558 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:48.425370    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddxxdqf" event={"ID":"ef15392d-8249-4cfc-a52b-7465b8219c96","Type":"ContainerStarted","Data":"233eaf4c4c508d321525cb45d35c2273ae5700544eedfb60a4ef1cafc6aaf8fe"}
Sep 18 20:32:48.425558 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:48.425404    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddxxdqf"
Sep 18 20:32:48.469524 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:48.469407    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddxxdqf" podStartSLOduration=1.469387614 podStartE2EDuration="1.469387614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:32:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:32:48.467329632 +0000 UTC m=+3284.977685724" watchObservedRunningTime="2026-09-18 20:32:48.469387614 +0000 UTC m=+3284.979743705"
Sep 18 20:32:49.235602 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:49.235563    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-8n4dh" podUID="6fb50972-c126-4a2c-9b1b-dedf882cd1b0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:32:50.246874 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:50.246833    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddxxdqf"]
Sep 18 20:32:50.259194 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:50.259154    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:32:50.327131 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:50.327093    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:32:50.447710 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:50.447675    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:32:52.978547 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:52.978508    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:32:53.080969 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:53.080937    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:32:53.179272 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:53.179234    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:32:54.458207 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:54.458172    2564 generic.go:356] "Generic (PLEG): container finished" podID="6fb50972-c126-4a2c-9b1b-dedf882cd1b0" containerID="fe0f0337106624b000fae30b51465fd2129f5f457ebfd6e3a384063426a60ae4" exitCode=0
Sep 18 20:32:54.458606 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:54.458255    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-8n4dh" event={"ID":"6fb50972-c126-4a2c-9b1b-dedf882cd1b0","Type":"ContainerDied","Data":"fe0f0337106624b000fae30b51465fd2129f5f457ebfd6e3a384063426a60ae4"}
Sep 18 20:32:54.514426 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:54.514348    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-8n4dh"
Sep 18 20:32:54.598743 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:54.598704    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xzn2s\" (UniqueName: \"kubernetes.io/projected/6fb50972-c126-4a2c-9b1b-dedf882cd1b0-kube-api-access-xzn2s\") pod \"6fb50972-c126-4a2c-9b1b-dedf882cd1b0\" (UID: \"6fb50972-c126-4a2c-9b1b-dedf882cd1b0\") "
Sep 18 20:32:54.598929 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:54.598914    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fb50972-c126-4a2c-9b1b-dedf882cd1b0-tls-certs\") pod \"6fb50972-c126-4a2c-9b1b-dedf882cd1b0\" (UID: \"6fb50972-c126-4a2c-9b1b-dedf882cd1b0\") "
Sep 18 20:32:54.601058 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:54.601026    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fb50972-c126-4a2c-9b1b-dedf882cd1b0-kube-api-access-xzn2s" (OuterVolumeSpecName: "kube-api-access-xzn2s") pod "6fb50972-c126-4a2c-9b1b-dedf882cd1b0" (UID: "6fb50972-c126-4a2c-9b1b-dedf882cd1b0"). InnerVolumeSpecName "kube-api-access-xzn2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:32:54.601178 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:54.601052    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fb50972-c126-4a2c-9b1b-dedf882cd1b0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6fb50972-c126-4a2c-9b1b-dedf882cd1b0" (UID: "6fb50972-c126-4a2c-9b1b-dedf882cd1b0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:32:54.700371 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:54.700331    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fb50972-c126-4a2c-9b1b-dedf882cd1b0-tls-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:32:54.700371 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:54.700363    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzn2s\" (UniqueName: \"kubernetes.io/projected/6fb50972-c126-4a2c-9b1b-dedf882cd1b0-kube-api-access-xzn2s\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:32:55.464666 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:55.464633    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-8n4dh" event={"ID":"6fb50972-c126-4a2c-9b1b-dedf882cd1b0","Type":"ContainerDied","Data":"070612c64b2c1325a86c71db19a0d76f790a819b1500faea0f842543f7a829d2"}
Sep 18 20:32:55.465172 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:55.464688    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-8n4dh"
Sep 18 20:32:55.465172 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:55.464697    2564 scope.go:117] "RemoveContainer" containerID="fe0f0337106624b000fae30b51465fd2129f5f457ebfd6e3a384063426a60ae4"
Sep 18 20:32:55.488037 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:55.488003    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-8n4dh"]
Sep 18 20:32:55.489559 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:55.489535    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-8n4dh"]
Sep 18 20:32:55.999771 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:55.999739    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6fb50972-c126-4a2c-9b1b-dedf882cd1b0" path="/var/lib/kubelet/pods/6fb50972-c126-4a2c-9b1b-dedf882cd1b0/volumes"
Sep 18 20:32:58.056821 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:32:58.056787    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:33:04.179195 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:33:04.179166    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-58vl4_945a444e-9381-4e2f-b4c1-64be0ec88e6c/console-operator/2.log"
Sep 18 20:33:04.189473 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:33:04.189440    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-58vl4_945a444e-9381-4e2f-b4c1-64be0ec88e6c/console-operator/2.log"
Sep 18 20:33:05.456103 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:33:05.456067    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddxxdqf" podUID="ef15392d-8249-4cfc-a52b-7465b8219c96" containerName="main" containerID="cri-o://7a16db9be3f6fe2f30e140d39ce1a5b786e18dc733163ceac543e781441a5759" gracePeriod=45
Sep 18 20:33:08.664586 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:33:08.664543    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:33:08.887105 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:33:08.887074    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-wx6bv"]
Sep 18 20:33:08.887471 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:33:08.887457    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fb50972-c126-4a2c-9b1b-dedf882cd1b0" containerName="main"
Sep 18 20:33:08.887536 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:33:08.887473    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fb50972-c126-4a2c-9b1b-dedf882cd1b0" containerName="main"
Sep 18 20:33:08.887573 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:33:08.887537    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fb50972-c126-4a2c-9b1b-dedf882cd1b0" containerName="main"
Sep 18 20:33:08.892008 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:33:08.891975    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-wx6bv"
Sep 18 20:33:08.894475 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:33:08.894438    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-ndbb9\""
Sep 18 20:33:08.894475 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:33:08.894438    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-g4sg4\""
Sep 18 20:33:08.894689 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:33:08.894516    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 18 20:33:08.894689 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:33:08.894543    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 18 20:33:08.894794 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:33:08.894749    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 18 20:33:08.898254 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:33:08.898221    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-wx6bv"]
Sep 18 20:33:08.922691 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:33:08.922597    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:33:09.020967 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:33:09.020930    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zgfr\" (UniqueName: \"kubernetes.io/projected/6c7383ee-4f9c-4281-a3d1-ae91b76157bc-kube-api-access-6zgfr\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-wx6bv\" (UID: \"6c7383ee-4f9c-4281-a3d1-ae91b76157bc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-wx6bv"
Sep 18 20:33:09.021177 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:33:09.021048    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c7383ee-4f9c-4281-a3d1-ae91b76157bc-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-wx6bv\" (UID: \"6c7383ee-4f9c-4281-a3d1-ae91b76157bc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-wx6bv"
Sep 18 20:33:09.122062 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:33:09.122013    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c7383ee-4f9c-4281-a3d1-ae91b76157bc-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-wx6bv\" (UID: \"6c7383ee-4f9c-4281-a3d1-ae91b76157bc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-wx6bv"
Sep 18 20:33:09.122256 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:33:09.122119    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zgfr\" (UniqueName: \"kubernetes.io/projected/6c7383ee-4f9c-4281-a3d1-ae91b76157bc-kube-api-access-6zgfr\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-wx6bv\" (UID: \"6c7383ee-4f9c-4281-a3d1-ae91b76157bc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-wx6bv"
Sep 18 20:33:09.124571 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:33:09.124549    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c7383ee-4f9c-4281-a3d1-ae91b76157bc-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-wx6bv\" (UID: \"6c7383ee-4f9c-4281-a3d1-ae91b76157bc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-wx6bv"
Sep 18 20:33:09.133644 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:33:09.133616    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zgfr\" (UniqueName: \"kubernetes.io/projected/6c7383ee-4f9c-4281-a3d1-ae91b76157bc-kube-api-access-6zgfr\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-wx6bv\" (UID: \"6c7383ee-4f9c-4281-a3d1-ae91b76157bc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-wx6bv"
Sep 18 20:33:09.205469 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:33:09.205368    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-wx6bv"
Sep 18 20:33:09.478547 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:33:09.478518    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-wx6bv"]
Sep 18 20:33:09.481552 ip-10-0-128-161 kubenswrapper[2564]: W0918 20:33:09.481510    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c7383ee_4f9c_4281_a3d1_ae91b76157bc.slice/crio-c79fcbe1a9c1fdb4be4d7984c50c4cc48b3464e5387494b42ae549e6c33bf2b4 WatchSource:0}: Error finding container c79fcbe1a9c1fdb4be4d7984c50c4cc48b3464e5387494b42ae549e6c33bf2b4: Status 404 returned error can't find the container with id c79fcbe1a9c1fdb4be4d7984c50c4cc48b3464e5387494b42ae549e6c33bf2b4
Sep 18 20:33:09.536591 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:33:09.536549    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-wx6bv" event={"ID":"6c7383ee-4f9c-4281-a3d1-ae91b76157bc","Type":"ContainerStarted","Data":"c79fcbe1a9c1fdb4be4d7984c50c4cc48b3464e5387494b42ae549e6c33bf2b4"}
Sep 18 20:33:10.542611 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:33:10.542568    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-wx6bv" event={"ID":"6c7383ee-4f9c-4281-a3d1-ae91b76157bc","Type":"ContainerStarted","Data":"9e31a769e731c923be235d136b89f37c506aa40983510ef617a9a4796353c4ed"}
Sep 18 20:33:10.543110 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:33:10.542689    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-wx6bv"
Sep 18 20:33:10.575381 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:33:10.575323    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-wx6bv" podStartSLOduration=2.575306896 podStartE2EDuration="2.575306896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:33:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:33:10.574090126 +0000 UTC m=+3307.084446504" watchObservedRunningTime="2026-09-18 20:33:10.575306896 +0000 UTC m=+3307.085662987"
Sep 18 20:33:16.695857 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:33:16.695820    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:33:20.440260 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:33:20.440217    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddxxdqf" podUID="ef15392d-8249-4cfc-a52b-7465b8219c96" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:33:22.859486 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:33:22.859400    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:33:30.437949 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:33:30.437849    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddxxdqf" podUID="ef15392d-8249-4cfc-a52b-7465b8219c96" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:33:32.258107 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:33:32.258058    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:33:32.313969 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:33:32.313931    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:33:35.668963 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:33:35.668927    2564 generic.go:356] "Generic (PLEG): container finished" podID="ef15392d-8249-4cfc-a52b-7465b8219c96" containerID="7a16db9be3f6fe2f30e140d39ce1a5b786e18dc733163ceac543e781441a5759" exitCode=0
Sep 18 20:33:35.669364 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:33:35.669010    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddxxdqf" event={"ID":"ef15392d-8249-4cfc-a52b-7465b8219c96","Type":"ContainerDied","Data":"7a16db9be3f6fe2f30e140d39ce1a5b786e18dc733163ceac543e781441a5759"}
Sep 18 20:33:35.696470 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:33:35.696443    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddxxdqf"
Sep 18 20:33:35.745745 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:33:35.745652    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c5v9b\" (UniqueName: \"kubernetes.io/projected/ef15392d-8249-4cfc-a52b-7465b8219c96-kube-api-access-c5v9b\") pod \"ef15392d-8249-4cfc-a52b-7465b8219c96\" (UID: \"ef15392d-8249-4cfc-a52b-7465b8219c96\") "
Sep 18 20:33:35.745745 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:33:35.745690    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef15392d-8249-4cfc-a52b-7465b8219c96-tls-certs\") pod \"ef15392d-8249-4cfc-a52b-7465b8219c96\" (UID: \"ef15392d-8249-4cfc-a52b-7465b8219c96\") "
Sep 18 20:33:35.748185 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:33:35.748149    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef15392d-8249-4cfc-a52b-7465b8219c96-kube-api-access-c5v9b" (OuterVolumeSpecName: "kube-api-access-c5v9b") pod "ef15392d-8249-4cfc-a52b-7465b8219c96" (UID: "ef15392d-8249-4cfc-a52b-7465b8219c96"). InnerVolumeSpecName "kube-api-access-c5v9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:33:35.748185 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:33:35.748173    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef15392d-8249-4cfc-a52b-7465b8219c96-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ef15392d-8249-4cfc-a52b-7465b8219c96" (UID: "ef15392d-8249-4cfc-a52b-7465b8219c96"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:33:35.846693 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:33:35.846653    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5v9b\" (UniqueName: \"kubernetes.io/projected/ef15392d-8249-4cfc-a52b-7465b8219c96-kube-api-access-c5v9b\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:33:35.846693 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:33:35.846687    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef15392d-8249-4cfc-a52b-7465b8219c96-tls-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:33:36.674943 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:33:36.674913    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddxxdqf"
Sep 18 20:33:36.675382 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:33:36.674899    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddxxdqf" event={"ID":"ef15392d-8249-4cfc-a52b-7465b8219c96","Type":"ContainerDied","Data":"233eaf4c4c508d321525cb45d35c2273ae5700544eedfb60a4ef1cafc6aaf8fe"}
Sep 18 20:33:36.675382 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:33:36.675092    2564 scope.go:117] "RemoveContainer" containerID="7a16db9be3f6fe2f30e140d39ce1a5b786e18dc733163ceac543e781441a5759"
Sep 18 20:33:36.690781 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:33:36.690747    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddxxdqf"]
Sep 18 20:33:36.692852 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:33:36.692827    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddxxdqf"]
Sep 18 20:33:38.000653 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:33:38.000614    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef15392d-8249-4cfc-a52b-7465b8219c96" path="/var/lib/kubelet/pods/ef15392d-8249-4cfc-a52b-7465b8219c96/volumes"
Sep 18 20:33:41.549882 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:33:41.549853    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-wx6bv"
Sep 18 20:34:02.528240 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:34:02.528205    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:34:05.893612 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:34:05.893571    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:34:21.308852 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:34:21.308816    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:34:22.334281 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:34:22.334246    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:34:22.436136 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:34:22.436097    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:34:22.710321 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:34:22.710280    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:34:22.808287 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:34:22.808227    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:34:25.984862 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:34:25.984824    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:34:28.760157 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:34:28.760100    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:34:34.342907 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:34:34.342862    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:34:34.408385 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:34:34.408344    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:34:37.199904 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:34:37.199865    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:34:37.351629 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:34:37.351587    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-2m7ml"]
Sep 18 20:34:37.351956 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:34:37.351944    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef15392d-8249-4cfc-a52b-7465b8219c96" containerName="main"
Sep 18 20:34:37.352028 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:34:37.351957    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef15392d-8249-4cfc-a52b-7465b8219c96" containerName="main"
Sep 18 20:34:37.352088 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:34:37.352078    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef15392d-8249-4cfc-a52b-7465b8219c96" containerName="main"
Sep 18 20:34:37.355309 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:34:37.355284    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-2m7ml"
Sep 18 20:34:37.357636 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:34:37.357607    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Sep 18 20:34:37.358445 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:34:37.358422    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-epp-sa-dockercfg-28q62\""
Sep 18 20:34:37.358570 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:34:37.358509    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-2v6xm\""
Sep 18 20:34:37.358570 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:34:37.358561    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-kserve-self-signed-certs\""
Sep 18 20:34:37.358699 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:34:37.358573    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Sep 18 20:34:37.367842 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:34:37.367819    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-2m7ml"]
Sep 18 20:34:37.385866 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:34:37.385825    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5671f9f2-aca9-48f2-baeb-5e0125f725d6-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-2m7ml\" (UID: \"5671f9f2-aca9-48f2-baeb-5e0125f725d6\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-2m7ml"
Sep 18 20:34:37.386020 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:34:37.385910    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb4t4\" (UniqueName: \"kubernetes.io/projected/5671f9f2-aca9-48f2-baeb-5e0125f725d6-kube-api-access-fb4t4\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-2m7ml\" (UID: \"5671f9f2-aca9-48f2-baeb-5e0125f725d6\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-2m7ml"
Sep 18 20:34:37.486822 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:34:37.486732    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb4t4\" (UniqueName: \"kubernetes.io/projected/5671f9f2-aca9-48f2-baeb-5e0125f725d6-kube-api-access-fb4t4\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-2m7ml\" (UID: \"5671f9f2-aca9-48f2-baeb-5e0125f725d6\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-2m7ml"
Sep 18 20:34:37.486822 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:34:37.486791    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5671f9f2-aca9-48f2-baeb-5e0125f725d6-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-2m7ml\" (UID: \"5671f9f2-aca9-48f2-baeb-5e0125f725d6\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-2m7ml"
Sep 18 20:34:37.489474 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:34:37.489447    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5671f9f2-aca9-48f2-baeb-5e0125f725d6-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-2m7ml\" (UID: \"5671f9f2-aca9-48f2-baeb-5e0125f725d6\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-2m7ml"
Sep 18 20:34:37.495363 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:34:37.495333    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb4t4\" (UniqueName: \"kubernetes.io/projected/5671f9f2-aca9-48f2-baeb-5e0125f725d6-kube-api-access-fb4t4\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-2m7ml\" (UID: \"5671f9f2-aca9-48f2-baeb-5e0125f725d6\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-2m7ml"
Sep 18 20:34:37.596859 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:34:37.596824    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:34:37.666586 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:34:37.666539    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-2m7ml"
Sep 18 20:34:37.700518 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:34:37.700486    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:34:37.937703 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:34:37.937677    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-2m7ml"]
Sep 18 20:34:37.940152 ip-10-0-128-161 kubenswrapper[2564]: W0918 20:34:37.940125    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5671f9f2_aca9_48f2_baeb_5e0125f725d6.slice/crio-d1505b72e112f4eaf2545e65329a3e6c017aa48bde364c8d7e5b9e10cc767ad9 WatchSource:0}: Error finding container d1505b72e112f4eaf2545e65329a3e6c017aa48bde364c8d7e5b9e10cc767ad9: Status 404 returned error can't find the container with id d1505b72e112f4eaf2545e65329a3e6c017aa48bde364c8d7e5b9e10cc767ad9
Sep 18 20:34:37.981220 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:34:37.981189    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-2m7ml" event={"ID":"5671f9f2-aca9-48f2-baeb-5e0125f725d6","Type":"ContainerStarted","Data":"d1505b72e112f4eaf2545e65329a3e6c017aa48bde364c8d7e5b9e10cc767ad9"}
Sep 18 20:34:38.993445 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:34:38.993297    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-2m7ml" event={"ID":"5671f9f2-aca9-48f2-baeb-5e0125f725d6","Type":"ContainerStarted","Data":"a422fd8fb6461616b747e42de75d64b638f71c2b450cc8f72cc056f4733d8b02"}
Sep 18 20:34:38.993871 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:34:38.993598    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-2m7ml"
Sep 18 20:34:39.009273 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:34:39.009193    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-2m7ml" podStartSLOduration=2.00917768 podStartE2EDuration="2.00917768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:34:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:34:39.008160604 +0000 UTC m=+3395.518516720" watchObservedRunningTime="2026-09-18 20:34:39.00917768 +0000 UTC m=+3395.519533815"
Sep 18 20:35:10.000309 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:35:10.000228    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-2m7ml"
Sep 18 20:35:11.273055 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:35:11.273015    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-wx6bv"]
Sep 18 20:35:11.275604 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:35:11.275529    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:35:11.486179 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:35:11.486144    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:35:11.598843 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:35:11.598752    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:35:21.549871 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:35:21.549829    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-wx6bv" podUID="6c7383ee-4f9c-4281-a3d1-ae91b76157bc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:35:26.293835 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:35:26.293764    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-wx6bv" podUID="6c7383ee-4f9c-4281-a3d1-ae91b76157bc" containerName="main" containerID="cri-o://9e31a769e731c923be235d136b89f37c506aa40983510ef617a9a4796353c4ed" gracePeriod=45
Sep 18 20:35:29.692974 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:35:29.692931    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:35:29.863879 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:35:29.863841    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:35:31.550024 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:35:31.549965    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-wx6bv" podUID="6c7383ee-4f9c-4281-a3d1-ae91b76157bc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:35:32.510575 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:35:32.510540    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:35:33.590515 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:35:33.590473    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:35:41.549330 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:35:41.549290    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-wx6bv" podUID="6c7383ee-4f9c-4281-a3d1-ae91b76157bc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:35:41.549999 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:35:41.549419    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-wx6bv"
Sep 18 20:35:44.151701 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:35:44.151663    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:35:51.549334 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:35:51.549292    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-wx6bv" podUID="6c7383ee-4f9c-4281-a3d1-ae91b76157bc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:35:56.546732 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:35:56.546703    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-wx6bv"
Sep 18 20:35:56.688427 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:35:56.688385    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6zgfr\" (UniqueName: \"kubernetes.io/projected/6c7383ee-4f9c-4281-a3d1-ae91b76157bc-kube-api-access-6zgfr\") pod \"6c7383ee-4f9c-4281-a3d1-ae91b76157bc\" (UID: \"6c7383ee-4f9c-4281-a3d1-ae91b76157bc\") "
Sep 18 20:35:56.688623 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:35:56.688482    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c7383ee-4f9c-4281-a3d1-ae91b76157bc-tls-certs\") pod \"6c7383ee-4f9c-4281-a3d1-ae91b76157bc\" (UID: \"6c7383ee-4f9c-4281-a3d1-ae91b76157bc\") "
Sep 18 20:35:56.690728 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:35:56.690699    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c7383ee-4f9c-4281-a3d1-ae91b76157bc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6c7383ee-4f9c-4281-a3d1-ae91b76157bc" (UID: "6c7383ee-4f9c-4281-a3d1-ae91b76157bc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:35:56.690857 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:35:56.690833    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c7383ee-4f9c-4281-a3d1-ae91b76157bc-kube-api-access-6zgfr" (OuterVolumeSpecName: "kube-api-access-6zgfr") pod "6c7383ee-4f9c-4281-a3d1-ae91b76157bc" (UID: "6c7383ee-4f9c-4281-a3d1-ae91b76157bc"). InnerVolumeSpecName "kube-api-access-6zgfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:35:56.789552 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:35:56.789507    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zgfr\" (UniqueName: \"kubernetes.io/projected/6c7383ee-4f9c-4281-a3d1-ae91b76157bc-kube-api-access-6zgfr\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:35:56.789552 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:35:56.789543    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c7383ee-4f9c-4281-a3d1-ae91b76157bc-tls-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:35:57.396513 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:35:57.396476    2564 generic.go:356] "Generic (PLEG): container finished" podID="6c7383ee-4f9c-4281-a3d1-ae91b76157bc" containerID="9e31a769e731c923be235d136b89f37c506aa40983510ef617a9a4796353c4ed" exitCode=0
Sep 18 20:35:57.396714 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:35:57.396521    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-wx6bv"
Sep 18 20:35:57.396714 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:35:57.396560    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-wx6bv" event={"ID":"6c7383ee-4f9c-4281-a3d1-ae91b76157bc","Type":"ContainerDied","Data":"9e31a769e731c923be235d136b89f37c506aa40983510ef617a9a4796353c4ed"}
Sep 18 20:35:57.396714 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:35:57.396615    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-wx6bv" event={"ID":"6c7383ee-4f9c-4281-a3d1-ae91b76157bc","Type":"ContainerDied","Data":"c79fcbe1a9c1fdb4be4d7984c50c4cc48b3464e5387494b42ae549e6c33bf2b4"}
Sep 18 20:35:57.396714 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:35:57.396644    2564 scope.go:117] "RemoveContainer" containerID="9e31a769e731c923be235d136b89f37c506aa40983510ef617a9a4796353c4ed"
Sep 18 20:35:57.405936 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:35:57.405914    2564 scope.go:117] "RemoveContainer" containerID="9e31a769e731c923be235d136b89f37c506aa40983510ef617a9a4796353c4ed"
Sep 18 20:35:57.406335 ip-10-0-128-161 kubenswrapper[2564]: E0918 20:35:57.406311    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e31a769e731c923be235d136b89f37c506aa40983510ef617a9a4796353c4ed\": container with ID starting with 9e31a769e731c923be235d136b89f37c506aa40983510ef617a9a4796353c4ed not found: ID does not exist" containerID="9e31a769e731c923be235d136b89f37c506aa40983510ef617a9a4796353c4ed"
Sep 18 20:35:57.406400 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:35:57.406347    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e31a769e731c923be235d136b89f37c506aa40983510ef617a9a4796353c4ed"} err="failed to get container status \"9e31a769e731c923be235d136b89f37c506aa40983510ef617a9a4796353c4ed\": rpc error: code = NotFound desc = could not find container \"9e31a769e731c923be235d136b89f37c506aa40983510ef617a9a4796353c4ed\": container with ID starting with 9e31a769e731c923be235d136b89f37c506aa40983510ef617a9a4796353c4ed not found: ID does not exist"
Sep 18 20:35:57.417587 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:35:57.417551    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-wx6bv"]
Sep 18 20:35:57.421228 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:35:57.421200    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-wx6bv"]
Sep 18 20:35:58.000126 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:35:58.000081    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c7383ee-4f9c-4281-a3d1-ae91b76157bc" path="/var/lib/kubelet/pods/6c7383ee-4f9c-4281-a3d1-ae91b76157bc/volumes"
Sep 18 20:37:12.858963 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:37:12.858921    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:37:31.565157 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:37:31.565117    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:37:31.631637 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:37:31.631600    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:38:04.211843 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:38:04.211816    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-58vl4_945a444e-9381-4e2f-b4c1-64be0ec88e6c/console-operator/2.log"
Sep 18 20:38:04.222017 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:38:04.221976    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-58vl4_945a444e-9381-4e2f-b4c1-64be0ec88e6c/console-operator/2.log"
Sep 18 20:38:21.651114 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:38:21.651078    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:38:21.751042 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:38:21.751010    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:38:21.852822 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:38:21.852787    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:38:28.060422 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:38:28.060384    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:38:33.492531 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:38:33.492494    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:38:33.592410 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:38:33.592362    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:39:01.975905 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:39:01.975866    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:39:02.031957 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:39:02.031917    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:39:09.983696 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:39:09.983657    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:40:36.169269 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:40:36.169217    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:40:36.198298 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:40:36.198262    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-2m7ml"]
Sep 18 20:40:36.252478 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:40:36.252437    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:40:39.999760 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:40:39.999723    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-2m7ml" podUID="5671f9f2-aca9-48f2-baeb-5e0125f725d6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:40:49.999772 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:40:49.999734    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-2m7ml" podUID="5671f9f2-aca9-48f2-baeb-5e0125f725d6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:40:51.217689 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:40:51.217640    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-2m7ml" podUID="5671f9f2-aca9-48f2-baeb-5e0125f725d6" containerName="main" containerID="cri-o://a422fd8fb6461616b747e42de75d64b638f71c2b450cc8f72cc056f4733d8b02" gracePeriod=45
Sep 18 20:40:55.712237 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:40:55.712198    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:40:59.999510 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:40:59.999463    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-2m7ml" podUID="5671f9f2-aca9-48f2-baeb-5e0125f725d6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:41:00.000336 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:41:00.000318    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-2m7ml"
Sep 18 20:41:09.999404 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:41:09.999366    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-2m7ml" podUID="5671f9f2-aca9-48f2-baeb-5e0125f725d6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:41:15.827832 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:41:15.827790    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:41:15.931152 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:41:15.931112    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:41:16.031297 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:41:16.031262    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:41:19.999241 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:41:19.999201    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-2m7ml" podUID="5671f9f2-aca9-48f2-baeb-5e0125f725d6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:41:21.481315 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:41:21.481255    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-2m7ml"
Sep 18 20:41:21.586696 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:41:21.586655    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5671f9f2-aca9-48f2-baeb-5e0125f725d6-tls-certs\") pod \"5671f9f2-aca9-48f2-baeb-5e0125f725d6\" (UID: \"5671f9f2-aca9-48f2-baeb-5e0125f725d6\") "
Sep 18 20:41:21.586874 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:41:21.586774    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fb4t4\" (UniqueName: \"kubernetes.io/projected/5671f9f2-aca9-48f2-baeb-5e0125f725d6-kube-api-access-fb4t4\") pod \"5671f9f2-aca9-48f2-baeb-5e0125f725d6\" (UID: \"5671f9f2-aca9-48f2-baeb-5e0125f725d6\") "
Sep 18 20:41:21.589073 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:41:21.589040    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5671f9f2-aca9-48f2-baeb-5e0125f725d6-kube-api-access-fb4t4" (OuterVolumeSpecName: "kube-api-access-fb4t4") pod "5671f9f2-aca9-48f2-baeb-5e0125f725d6" (UID: "5671f9f2-aca9-48f2-baeb-5e0125f725d6"). InnerVolumeSpecName "kube-api-access-fb4t4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:41:21.589073 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:41:21.589067    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5671f9f2-aca9-48f2-baeb-5e0125f725d6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5671f9f2-aca9-48f2-baeb-5e0125f725d6" (UID: "5671f9f2-aca9-48f2-baeb-5e0125f725d6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:41:21.688097 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:41:21.688064    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fb4t4\" (UniqueName: \"kubernetes.io/projected/5671f9f2-aca9-48f2-baeb-5e0125f725d6-kube-api-access-fb4t4\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:41:21.688097 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:41:21.688093    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5671f9f2-aca9-48f2-baeb-5e0125f725d6-tls-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:41:21.988695 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:41:21.988660    2564 generic.go:356] "Generic (PLEG): container finished" podID="5671f9f2-aca9-48f2-baeb-5e0125f725d6" containerID="a422fd8fb6461616b747e42de75d64b638f71c2b450cc8f72cc056f4733d8b02" exitCode=0
Sep 18 20:41:21.988858 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:41:21.988704    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-2m7ml"
Sep 18 20:41:21.988858 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:41:21.988738    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-2m7ml" event={"ID":"5671f9f2-aca9-48f2-baeb-5e0125f725d6","Type":"ContainerDied","Data":"a422fd8fb6461616b747e42de75d64b638f71c2b450cc8f72cc056f4733d8b02"}
Sep 18 20:41:21.988858 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:41:21.988794    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-2m7ml" event={"ID":"5671f9f2-aca9-48f2-baeb-5e0125f725d6","Type":"ContainerDied","Data":"d1505b72e112f4eaf2545e65329a3e6c017aa48bde364c8d7e5b9e10cc767ad9"}
Sep 18 20:41:21.988858 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:41:21.988817    2564 scope.go:117] "RemoveContainer" containerID="a422fd8fb6461616b747e42de75d64b638f71c2b450cc8f72cc056f4733d8b02"
Sep 18 20:41:21.998264 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:41:21.998247    2564 scope.go:117] "RemoveContainer" containerID="a422fd8fb6461616b747e42de75d64b638f71c2b450cc8f72cc056f4733d8b02"
Sep 18 20:41:21.998535 ip-10-0-128-161 kubenswrapper[2564]: E0918 20:41:21.998514    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a422fd8fb6461616b747e42de75d64b638f71c2b450cc8f72cc056f4733d8b02\": container with ID starting with a422fd8fb6461616b747e42de75d64b638f71c2b450cc8f72cc056f4733d8b02 not found: ID does not exist" containerID="a422fd8fb6461616b747e42de75d64b638f71c2b450cc8f72cc056f4733d8b02"
Sep 18 20:41:21.998604 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:41:21.998539    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a422fd8fb6461616b747e42de75d64b638f71c2b450cc8f72cc056f4733d8b02"} err="failed to get container status \"a422fd8fb6461616b747e42de75d64b638f71c2b450cc8f72cc056f4733d8b02\": rpc error: code = NotFound desc = could not find container \"a422fd8fb6461616b747e42de75d64b638f71c2b450cc8f72cc056f4733d8b02\": container with ID starting with a422fd8fb6461616b747e42de75d64b638f71c2b450cc8f72cc056f4733d8b02 not found: ID does not exist"
Sep 18 20:41:22.012738 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:41:22.012702    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-2m7ml"]
Sep 18 20:41:22.016280 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:41:22.016246    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-2m7ml"]
Sep 18 20:41:22.024641 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:41:22.024615    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:41:22.084215 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:41:22.084177    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:41:24.000471 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:41:24.000436    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5671f9f2-aca9-48f2-baeb-5e0125f725d6" path="/var/lib/kubelet/pods/5671f9f2-aca9-48f2-baeb-5e0125f725d6/volumes"
Sep 18 20:41:26.264551 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:41:26.264491    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:41:26.360835 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:41:26.360798    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:41:26.888111 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:41:26.888078    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:41:31.313868 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:41:31.313834    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:41:31.413791 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:41:31.413746    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:41:36.866105 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:41:36.865974    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:41:36.933445 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:41:36.933408    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:41:42.106288 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:41:42.106249    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:41:42.299319 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:41:42.299272    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schetx4hz"]
Sep 18 20:41:42.299753 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:41:42.299731    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c7383ee-4f9c-4281-a3d1-ae91b76157bc" containerName="main"
Sep 18 20:41:42.299843 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:41:42.299756    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c7383ee-4f9c-4281-a3d1-ae91b76157bc" containerName="main"
Sep 18 20:41:42.299843 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:41:42.299772    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5671f9f2-aca9-48f2-baeb-5e0125f725d6" containerName="main"
Sep 18 20:41:42.299843 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:41:42.299779    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5671f9f2-aca9-48f2-baeb-5e0125f725d6" containerName="main"
Sep 18 20:41:42.300038 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:41:42.299871    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c7383ee-4f9c-4281-a3d1-ae91b76157bc" containerName="main"
Sep 18 20:41:42.300038 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:41:42.299884    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5671f9f2-aca9-48f2-baeb-5e0125f725d6" containerName="main"
Sep 18 20:41:42.304361 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:41:42.304336    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schetx4hz"
Sep 18 20:41:42.307244 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:41:42.307218    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 18 20:41:42.307371 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:41:42.307259    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-epp-sa-dockercfg-9kqkf\""
Sep 18 20:41:42.307762 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:41:42.307741    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 18 20:41:42.307861 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:41:42.307769    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-smjj5\""
Sep 18 20:41:42.307861 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:41:42.307769    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 18 20:41:42.314808 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:41:42.314785    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schetx4hz"]
Sep 18 20:41:42.379150 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:41:42.379043    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ab52f1e-8fb0-443a-805c-e82e906c0b54-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schetx4hz\" (UID: \"5ab52f1e-8fb0-443a-805c-e82e906c0b54\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schetx4hz"
Sep 18 20:41:42.379150 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:41:42.379105    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhvtn\" (UniqueName: \"kubernetes.io/projected/5ab52f1e-8fb0-443a-805c-e82e906c0b54-kube-api-access-zhvtn\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schetx4hz\" (UID: \"5ab52f1e-8fb0-443a-805c-e82e906c0b54\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schetx4hz"
Sep 18 20:41:42.480062 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:41:42.480023    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ab52f1e-8fb0-443a-805c-e82e906c0b54-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schetx4hz\" (UID: \"5ab52f1e-8fb0-443a-805c-e82e906c0b54\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schetx4hz"
Sep 18 20:41:42.480247 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:41:42.480107    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhvtn\" (UniqueName: \"kubernetes.io/projected/5ab52f1e-8fb0-443a-805c-e82e906c0b54-kube-api-access-zhvtn\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schetx4hz\" (UID: \"5ab52f1e-8fb0-443a-805c-e82e906c0b54\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schetx4hz"
Sep 18 20:41:42.482592 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:41:42.482564    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ab52f1e-8fb0-443a-805c-e82e906c0b54-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schetx4hz\" (UID: \"5ab52f1e-8fb0-443a-805c-e82e906c0b54\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schetx4hz"
Sep 18 20:41:42.488300 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:41:42.488267    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhvtn\" (UniqueName: \"kubernetes.io/projected/5ab52f1e-8fb0-443a-805c-e82e906c0b54-kube-api-access-zhvtn\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schetx4hz\" (UID: \"5ab52f1e-8fb0-443a-805c-e82e906c0b54\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schetx4hz"
Sep 18 20:41:42.588238 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:41:42.588198    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:41:42.615862 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:41:42.615827    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schetx4hz"
Sep 18 20:41:42.893369 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:41:42.893341    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schetx4hz"]
Sep 18 20:41:42.896042 ip-10-0-128-161 kubenswrapper[2564]: W0918 20:41:42.896010    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ab52f1e_8fb0_443a_805c_e82e906c0b54.slice/crio-e5ea8e79f7b8afa6743f5c8a3433174083987e19baf3b2532337aba4ac0c5f3d WatchSource:0}: Error finding container e5ea8e79f7b8afa6743f5c8a3433174083987e19baf3b2532337aba4ac0c5f3d: Status 404 returned error can't find the container with id e5ea8e79f7b8afa6743f5c8a3433174083987e19baf3b2532337aba4ac0c5f3d
Sep 18 20:41:43.093947 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:41:43.093909    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schetx4hz" event={"ID":"5ab52f1e-8fb0-443a-805c-e82e906c0b54","Type":"ContainerStarted","Data":"874c49e531d1c5a278cb5f420d42d8c8d8167fc48ddf0b714e77c40106123e05"}
Sep 18 20:41:43.094125 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:41:43.093974    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schetx4hz" event={"ID":"5ab52f1e-8fb0-443a-805c-e82e906c0b54","Type":"ContainerStarted","Data":"e5ea8e79f7b8afa6743f5c8a3433174083987e19baf3b2532337aba4ac0c5f3d"}
Sep 18 20:41:43.094125 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:41:43.094049    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schetx4hz"
Sep 18 20:41:43.109811 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:41:43.109758    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schetx4hz" podStartSLOduration=1.109741553 podStartE2EDuration="1.109741553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:41:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:41:43.10833283 +0000 UTC m=+3819.618688938" watchObservedRunningTime="2026-09-18 20:41:43.109741553 +0000 UTC m=+3819.620097644"
Sep 18 20:41:53.140292 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:41:53.140250    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:41:53.905289 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:41:53.905250    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:41:57.793861 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:41:57.793823    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:41:59.447473 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:41:59.447433    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:42:14.101121 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:42:14.101090    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schetx4hz"
Sep 18 20:42:19.356881 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:42:19.356840    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:42:22.443814 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:42:22.443773    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:42:22.541475 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:42:22.541436    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:43:04.241578 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:43:04.241550    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-58vl4_945a444e-9381-4e2f-b4c1-64be0ec88e6c/console-operator/2.log"
Sep 18 20:43:04.252080 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:43:04.252053    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-58vl4_945a444e-9381-4e2f-b4c1-64be0ec88e6c/console-operator/2.log"
Sep 18 20:43:39.745707 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:43:39.745634    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:43:59.659956 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:43:59.659835    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:43:59.751192 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:43:59.751149    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:44:05.588497 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:44:05.588457    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:44:05.668044 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:44:05.668000    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:44:09.235766 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:44:09.235726    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:44:09.416915 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:44:09.416877    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-prcgc"]
Sep 18 20:44:09.420507 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:44:09.420486    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-prcgc"
Sep 18 20:44:09.423801 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:44:09.423773    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-ww85g\""
Sep 18 20:44:09.424136 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:44:09.423791    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Sep 18 20:44:09.424332 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:44:09.423817    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-epp-sa-dockercfg-54fgf\""
Sep 18 20:44:09.424633 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:44:09.423845    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Sep 18 20:44:09.424791 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:44:09.423876    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Sep 18 20:44:09.430596 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:44:09.430569    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-prcgc"]
Sep 18 20:44:09.451453 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:44:09.451418    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:44:09.495679 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:44:09.495585    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/629401de-eb90-456c-a81b-ea5faf3561d9-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-prcgc\" (UID: \"629401de-eb90-456c-a81b-ea5faf3561d9\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-prcgc"
Sep 18 20:44:09.495679 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:44:09.495660    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmxrk\" (UniqueName: \"kubernetes.io/projected/629401de-eb90-456c-a81b-ea5faf3561d9-kube-api-access-jmxrk\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-prcgc\" (UID: \"629401de-eb90-456c-a81b-ea5faf3561d9\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-prcgc"
Sep 18 20:44:09.596398 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:44:09.596361    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/629401de-eb90-456c-a81b-ea5faf3561d9-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-prcgc\" (UID: \"629401de-eb90-456c-a81b-ea5faf3561d9\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-prcgc"
Sep 18 20:44:09.596575 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:44:09.596423    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmxrk\" (UniqueName: \"kubernetes.io/projected/629401de-eb90-456c-a81b-ea5faf3561d9-kube-api-access-jmxrk\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-prcgc\" (UID: \"629401de-eb90-456c-a81b-ea5faf3561d9\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-prcgc"
Sep 18 20:44:09.599064 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:44:09.599029    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/629401de-eb90-456c-a81b-ea5faf3561d9-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-prcgc\" (UID: \"629401de-eb90-456c-a81b-ea5faf3561d9\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-prcgc"
Sep 18 20:44:09.604617 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:44:09.604585    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmxrk\" (UniqueName: \"kubernetes.io/projected/629401de-eb90-456c-a81b-ea5faf3561d9-kube-api-access-jmxrk\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-prcgc\" (UID: \"629401de-eb90-456c-a81b-ea5faf3561d9\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-prcgc"
Sep 18 20:44:09.733502 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:44:09.733462    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-prcgc"
Sep 18 20:44:10.010083 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:44:10.010039    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-prcgc"]
Sep 18 20:44:10.011535 ip-10-0-128-161 kubenswrapper[2564]: W0918 20:44:10.011492    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod629401de_eb90_456c_a81b_ea5faf3561d9.slice/crio-639e48bf4aafb5f4252bbd1d05c268e9b4b1f284629dbe9dde885546cb1098e9 WatchSource:0}: Error finding container 639e48bf4aafb5f4252bbd1d05c268e9b4b1f284629dbe9dde885546cb1098e9: Status 404 returned error can't find the container with id 639e48bf4aafb5f4252bbd1d05c268e9b4b1f284629dbe9dde885546cb1098e9
Sep 18 20:44:10.823943 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:44:10.823908    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-prcgc" event={"ID":"629401de-eb90-456c-a81b-ea5faf3561d9","Type":"ContainerStarted","Data":"796eeb37073abcedeb0373876009fa3063a406440099561e57ac00575cedd23d"}
Sep 18 20:44:10.824480 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:44:10.823968    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-prcgc" event={"ID":"629401de-eb90-456c-a81b-ea5faf3561d9","Type":"ContainerStarted","Data":"639e48bf4aafb5f4252bbd1d05c268e9b4b1f284629dbe9dde885546cb1098e9"}
Sep 18 20:44:10.824480 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:44:10.824025    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-prcgc"
Sep 18 20:44:10.839131 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:44:10.839068    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-prcgc" podStartSLOduration=1.839048668 podStartE2EDuration="1.839048668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:44:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:44:10.838896845 +0000 UTC m=+3967.349252957" watchObservedRunningTime="2026-09-18 20:44:10.839048668 +0000 UTC m=+3967.349404760"
Sep 18 20:44:18.050200 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:44:18.050159    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:44:21.551403 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:44:21.551363    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schetx4hz"]
Sep 18 20:44:21.565647 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:44:21.565609    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:44:21.656597 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:44:21.656563    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:44:24.101366 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:44:24.101313    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schetx4hz" podUID="5ab52f1e-8fb0-443a-805c-e82e906c0b54" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:44:34.101279 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:44:34.101231    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schetx4hz" podUID="5ab52f1e-8fb0-443a-805c-e82e906c0b54" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:44:36.573136 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:44:36.573075    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schetx4hz" podUID="5ab52f1e-8fb0-443a-805c-e82e906c0b54" containerName="main" containerID="cri-o://874c49e531d1c5a278cb5f420d42d8c8d8167fc48ddf0b714e77c40106123e05" gracePeriod=45
Sep 18 20:44:41.830808 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:44:41.830773    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-prcgc"
Sep 18 20:44:44.101571 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:44:44.101531    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schetx4hz" podUID="5ab52f1e-8fb0-443a-805c-e82e906c0b54" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:44:44.101952 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:44:44.101651    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schetx4hz"
Sep 18 20:44:54.101419 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:44:54.101376    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schetx4hz" podUID="5ab52f1e-8fb0-443a-805c-e82e906c0b54" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:45:04.101093 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:45:04.101048    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schetx4hz" podUID="5ab52f1e-8fb0-443a-805c-e82e906c0b54" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:45:06.844527 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:45:06.844454    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schetx4hz"
Sep 18 20:45:06.919684 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:45:06.919642    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ab52f1e-8fb0-443a-805c-e82e906c0b54-tls-certs\") pod \"5ab52f1e-8fb0-443a-805c-e82e906c0b54\" (UID: \"5ab52f1e-8fb0-443a-805c-e82e906c0b54\") "
Sep 18 20:45:06.919908 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:45:06.919754    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zhvtn\" (UniqueName: \"kubernetes.io/projected/5ab52f1e-8fb0-443a-805c-e82e906c0b54-kube-api-access-zhvtn\") pod \"5ab52f1e-8fb0-443a-805c-e82e906c0b54\" (UID: \"5ab52f1e-8fb0-443a-805c-e82e906c0b54\") "
Sep 18 20:45:06.922220 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:45:06.922185    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ab52f1e-8fb0-443a-805c-e82e906c0b54-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5ab52f1e-8fb0-443a-805c-e82e906c0b54" (UID: "5ab52f1e-8fb0-443a-805c-e82e906c0b54"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:45:06.922393 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:45:06.922246    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ab52f1e-8fb0-443a-805c-e82e906c0b54-kube-api-access-zhvtn" (OuterVolumeSpecName: "kube-api-access-zhvtn") pod "5ab52f1e-8fb0-443a-805c-e82e906c0b54" (UID: "5ab52f1e-8fb0-443a-805c-e82e906c0b54"). InnerVolumeSpecName "kube-api-access-zhvtn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:45:07.020611 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:45:07.020564    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ab52f1e-8fb0-443a-805c-e82e906c0b54-tls-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:45:07.020611 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:45:07.020598    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhvtn\" (UniqueName: \"kubernetes.io/projected/5ab52f1e-8fb0-443a-805c-e82e906c0b54-kube-api-access-zhvtn\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:45:07.116410 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:45:07.116322    2564 generic.go:356] "Generic (PLEG): container finished" podID="5ab52f1e-8fb0-443a-805c-e82e906c0b54" containerID="874c49e531d1c5a278cb5f420d42d8c8d8167fc48ddf0b714e77c40106123e05" exitCode=0
Sep 18 20:45:07.116410 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:45:07.116373    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schetx4hz" event={"ID":"5ab52f1e-8fb0-443a-805c-e82e906c0b54","Type":"ContainerDied","Data":"874c49e531d1c5a278cb5f420d42d8c8d8167fc48ddf0b714e77c40106123e05"}
Sep 18 20:45:07.116605 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:45:07.116413    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schetx4hz" event={"ID":"5ab52f1e-8fb0-443a-805c-e82e906c0b54","Type":"ContainerDied","Data":"e5ea8e79f7b8afa6743f5c8a3433174083987e19baf3b2532337aba4ac0c5f3d"}
Sep 18 20:45:07.116605 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:45:07.116434    2564 scope.go:117] "RemoveContainer" containerID="874c49e531d1c5a278cb5f420d42d8c8d8167fc48ddf0b714e77c40106123e05"
Sep 18 20:45:07.116605 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:45:07.116373    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schetx4hz"
Sep 18 20:45:07.125694 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:45:07.125671    2564 scope.go:117] "RemoveContainer" containerID="874c49e531d1c5a278cb5f420d42d8c8d8167fc48ddf0b714e77c40106123e05"
Sep 18 20:45:07.126045 ip-10-0-128-161 kubenswrapper[2564]: E0918 20:45:07.126014    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"874c49e531d1c5a278cb5f420d42d8c8d8167fc48ddf0b714e77c40106123e05\": container with ID starting with 874c49e531d1c5a278cb5f420d42d8c8d8167fc48ddf0b714e77c40106123e05 not found: ID does not exist" containerID="874c49e531d1c5a278cb5f420d42d8c8d8167fc48ddf0b714e77c40106123e05"
Sep 18 20:45:07.126149 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:45:07.126049    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"874c49e531d1c5a278cb5f420d42d8c8d8167fc48ddf0b714e77c40106123e05"} err="failed to get container status \"874c49e531d1c5a278cb5f420d42d8c8d8167fc48ddf0b714e77c40106123e05\": rpc error: code = NotFound desc = could not find container \"874c49e531d1c5a278cb5f420d42d8c8d8167fc48ddf0b714e77c40106123e05\": container with ID starting with 874c49e531d1c5a278cb5f420d42d8c8d8167fc48ddf0b714e77c40106123e05 not found: ID does not exist"
Sep 18 20:45:07.137362 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:45:07.137319    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schetx4hz"]
Sep 18 20:45:07.141046 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:45:07.141012    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schetx4hz"]
Sep 18 20:45:08.000863 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:45:08.000826    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ab52f1e-8fb0-443a-805c-e82e906c0b54" path="/var/lib/kubelet/pods/5ab52f1e-8fb0-443a-805c-e82e906c0b54/volumes"
Sep 18 20:45:11.349508 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:45:11.349468    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:45:11.445483 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:45:11.445445    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:45:17.828212 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:45:17.828169    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:45:23.407511 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:45:23.407466    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:45:23.468470 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:45:23.468429    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:45:24.545161 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:45:24.545104    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:45:24.728132 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:45:24.728096    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48mdlcg"]
Sep 18 20:45:24.728634 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:45:24.728607    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ab52f1e-8fb0-443a-805c-e82e906c0b54" containerName="main"
Sep 18 20:45:24.728634 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:45:24.728631    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ab52f1e-8fb0-443a-805c-e82e906c0b54" containerName="main"
Sep 18 20:45:24.728826 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:45:24.728730    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ab52f1e-8fb0-443a-805c-e82e906c0b54" containerName="main"
Sep 18 20:45:24.732028 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:45:24.732002    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48mdlcg"
Sep 18 20:45:24.734364 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:45:24.734335    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 18 20:45:24.734491 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:45:24.734384    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-zt8ws\""
Sep 18 20:45:24.735051 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:45:24.735027    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 18 20:45:24.735155 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:45:24.735118    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-epp-sa-dockercfg-9j6v8\""
Sep 18 20:45:24.735224 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:45:24.735211    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 18 20:45:24.744054 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:45:24.744025    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48mdlcg"]
Sep 18 20:45:24.757688 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:45:24.757656    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:45:24.785116 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:45:24.785077    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/439322c3-d40b-4902-83e7-1a6eaf95e6c8-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48mdlcg\" (UID: \"439322c3-d40b-4902-83e7-1a6eaf95e6c8\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48mdlcg"
Sep 18 20:45:24.785296 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:45:24.785156    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgkg5\" (UniqueName: \"kubernetes.io/projected/439322c3-d40b-4902-83e7-1a6eaf95e6c8-kube-api-access-dgkg5\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48mdlcg\" (UID: \"439322c3-d40b-4902-83e7-1a6eaf95e6c8\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48mdlcg"
Sep 18 20:45:24.886048 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:45:24.886011    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/439322c3-d40b-4902-83e7-1a6eaf95e6c8-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48mdlcg\" (UID: \"439322c3-d40b-4902-83e7-1a6eaf95e6c8\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48mdlcg"
Sep 18 20:45:24.886220 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:45:24.886071    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgkg5\" (UniqueName: \"kubernetes.io/projected/439322c3-d40b-4902-83e7-1a6eaf95e6c8-kube-api-access-dgkg5\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48mdlcg\" (UID: \"439322c3-d40b-4902-83e7-1a6eaf95e6c8\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48mdlcg"
Sep 18 20:45:24.888751 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:45:24.888725    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/439322c3-d40b-4902-83e7-1a6eaf95e6c8-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48mdlcg\" (UID: \"439322c3-d40b-4902-83e7-1a6eaf95e6c8\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48mdlcg"
Sep 18 20:45:24.893072 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:45:24.893047    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgkg5\" (UniqueName: \"kubernetes.io/projected/439322c3-d40b-4902-83e7-1a6eaf95e6c8-kube-api-access-dgkg5\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48mdlcg\" (UID: \"439322c3-d40b-4902-83e7-1a6eaf95e6c8\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48mdlcg"
Sep 18 20:45:25.043129 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:45:25.043099    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48mdlcg"
Sep 18 20:45:25.317155 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:45:25.317064    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48mdlcg"]
Sep 18 20:45:25.319953 ip-10-0-128-161 kubenswrapper[2564]: W0918 20:45:25.319923    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod439322c3_d40b_4902_83e7_1a6eaf95e6c8.slice/crio-439cdf8e3509f794cd6ee980bb9b7fcbb2f43b665053473d6943553f8cd876a6 WatchSource:0}: Error finding container 439cdf8e3509f794cd6ee980bb9b7fcbb2f43b665053473d6943553f8cd876a6: Status 404 returned error can't find the container with id 439cdf8e3509f794cd6ee980bb9b7fcbb2f43b665053473d6943553f8cd876a6
Sep 18 20:45:26.212437 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:45:26.212352    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48mdlcg" event={"ID":"439322c3-d40b-4902-83e7-1a6eaf95e6c8","Type":"ContainerStarted","Data":"f01eb149038f3a59906d28d3157d2d52de176a915cf77243ed71de46b8a753f0"}
Sep 18 20:45:26.212835 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:45:26.212439    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48mdlcg" event={"ID":"439322c3-d40b-4902-83e7-1a6eaf95e6c8","Type":"ContainerStarted","Data":"439cdf8e3509f794cd6ee980bb9b7fcbb2f43b665053473d6943553f8cd876a6"}
Sep 18 20:45:26.212835 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:45:26.212467    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48mdlcg"
Sep 18 20:45:26.227316 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:45:26.227168    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48mdlcg" podStartSLOduration=2.227148154 podStartE2EDuration="2.227148154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:45:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:45:26.226883117 +0000 UTC m=+4042.737239228" watchObservedRunningTime="2026-09-18 20:45:26.227148154 +0000 UTC m=+4042.737504245"
Sep 18 20:45:45.888269 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:45:45.888231    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:45:57.221664 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:45:57.221632    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48mdlcg"
Sep 18 20:46:14.528872 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:46:14.528834    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-prcgc"]
Sep 18 20:46:14.543399 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:46:14.543348    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:46:14.703571 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:46:14.703524    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:46:21.830813 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:46:21.830766    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-prcgc" podUID="629401de-eb90-456c-a81b-ea5faf3561d9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:46:29.549383 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:46:29.549307    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-prcgc" podUID="629401de-eb90-456c-a81b-ea5faf3561d9" containerName="main" containerID="cri-o://796eeb37073abcedeb0373876009fa3063a406440099561e57ac00575cedd23d" gracePeriod=45
Sep 18 20:46:31.831014 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:46:31.830952    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-prcgc" podUID="629401de-eb90-456c-a81b-ea5faf3561d9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:46:41.830818 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:46:41.830780    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-prcgc" podUID="629401de-eb90-456c-a81b-ea5faf3561d9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:46:41.831234 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:46:41.830887    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-prcgc"
Sep 18 20:46:51.830855 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:46:51.830814    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-prcgc" podUID="629401de-eb90-456c-a81b-ea5faf3561d9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:46:59.673747 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:46:59.673712    2564 generic.go:356] "Generic (PLEG): container finished" podID="629401de-eb90-456c-a81b-ea5faf3561d9" containerID="796eeb37073abcedeb0373876009fa3063a406440099561e57ac00575cedd23d" exitCode=0
Sep 18 20:46:59.674182 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:46:59.673779    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-prcgc" event={"ID":"629401de-eb90-456c-a81b-ea5faf3561d9","Type":"ContainerDied","Data":"796eeb37073abcedeb0373876009fa3063a406440099561e57ac00575cedd23d"}
Sep 18 20:46:59.809674 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:46:59.809647    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-prcgc"
Sep 18 20:46:59.831255 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:46:59.831159    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/629401de-eb90-456c-a81b-ea5faf3561d9-tls-certs\") pod \"629401de-eb90-456c-a81b-ea5faf3561d9\" (UID: \"629401de-eb90-456c-a81b-ea5faf3561d9\") "
Sep 18 20:46:59.831255 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:46:59.831250    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jmxrk\" (UniqueName: \"kubernetes.io/projected/629401de-eb90-456c-a81b-ea5faf3561d9-kube-api-access-jmxrk\") pod \"629401de-eb90-456c-a81b-ea5faf3561d9\" (UID: \"629401de-eb90-456c-a81b-ea5faf3561d9\") "
Sep 18 20:46:59.833636 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:46:59.833601    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/629401de-eb90-456c-a81b-ea5faf3561d9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "629401de-eb90-456c-a81b-ea5faf3561d9" (UID: "629401de-eb90-456c-a81b-ea5faf3561d9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:46:59.833636 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:46:59.833615    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/629401de-eb90-456c-a81b-ea5faf3561d9-kube-api-access-jmxrk" (OuterVolumeSpecName: "kube-api-access-jmxrk") pod "629401de-eb90-456c-a81b-ea5faf3561d9" (UID: "629401de-eb90-456c-a81b-ea5faf3561d9"). InnerVolumeSpecName "kube-api-access-jmxrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:46:59.932208 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:46:59.932167    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmxrk\" (UniqueName: \"kubernetes.io/projected/629401de-eb90-456c-a81b-ea5faf3561d9-kube-api-access-jmxrk\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:46:59.932208 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:46:59.932200    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/629401de-eb90-456c-a81b-ea5faf3561d9-tls-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:47:00.679625 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:47:00.679581    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-prcgc" event={"ID":"629401de-eb90-456c-a81b-ea5faf3561d9","Type":"ContainerDied","Data":"639e48bf4aafb5f4252bbd1d05c268e9b4b1f284629dbe9dde885546cb1098e9"}
Sep 18 20:47:00.679625 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:47:00.679626    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-prcgc"
Sep 18 20:47:00.680172 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:47:00.679654    2564 scope.go:117] "RemoveContainer" containerID="796eeb37073abcedeb0373876009fa3063a406440099561e57ac00575cedd23d"
Sep 18 20:47:00.694361 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:47:00.694325    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-prcgc"]
Sep 18 20:47:00.700345 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:47:00.700312    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-prcgc"]
Sep 18 20:47:02.000204 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:47:02.000168    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="629401de-eb90-456c-a81b-ea5faf3561d9" path="/var/lib/kubelet/pods/629401de-eb90-456c-a81b-ea5faf3561d9/volumes"
Sep 18 20:47:03.439522 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:47:03.439483    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:47:04.681774 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:47:04.681735    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:47:04.771930 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:47:04.771890    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:47:10.836237 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:47:10.836198    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:47:16.349149 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:47:16.349108    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:47:16.447329 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:47:16.447291    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:47:19.760932 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:47:19.760890    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:47:19.981532 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:47:19.981494    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-4257c"]
Sep 18 20:47:19.981890 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:47:19.981875    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="629401de-eb90-456c-a81b-ea5faf3561d9" containerName="main"
Sep 18 20:47:19.981890 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:47:19.981890    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="629401de-eb90-456c-a81b-ea5faf3561d9" containerName="main"
Sep 18 20:47:19.982121 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:47:19.981975    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="629401de-eb90-456c-a81b-ea5faf3561d9" containerName="main"
Sep 18 20:47:19.987086 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:47:19.987048    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-4257c"
Sep 18 20:47:19.990422 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:47:19.990393    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 18 20:47:19.990595 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:47:19.990393    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-gfj9n\""
Sep 18 20:47:19.990595 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:47:19.990393    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 18 20:47:19.990730 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:47:19.990394    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 18 20:47:19.990730 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:47:19.990510    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-epp-sa-dockercfg-pw5mw\""
Sep 18 20:47:20.004556 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:47:20.004525    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-4257c"]
Sep 18 20:47:20.032425 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:47:20.032329    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:47:20.106970 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:47:20.106930    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34ddeadc-2425-47fa-81d8-887f3488e6bf-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-4257c\" (UID: \"34ddeadc-2425-47fa-81d8-887f3488e6bf\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-4257c"
Sep 18 20:47:20.107197 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:47:20.107021    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wt48\" (UniqueName: \"kubernetes.io/projected/34ddeadc-2425-47fa-81d8-887f3488e6bf-kube-api-access-8wt48\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-4257c\" (UID: \"34ddeadc-2425-47fa-81d8-887f3488e6bf\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-4257c"
Sep 18 20:47:20.208293 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:47:20.208252    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wt48\" (UniqueName: \"kubernetes.io/projected/34ddeadc-2425-47fa-81d8-887f3488e6bf-kube-api-access-8wt48\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-4257c\" (UID: \"34ddeadc-2425-47fa-81d8-887f3488e6bf\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-4257c"
Sep 18 20:47:20.208471 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:47:20.208343    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34ddeadc-2425-47fa-81d8-887f3488e6bf-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-4257c\" (UID: \"34ddeadc-2425-47fa-81d8-887f3488e6bf\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-4257c"
Sep 18 20:47:20.210881 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:47:20.210850    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34ddeadc-2425-47fa-81d8-887f3488e6bf-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-4257c\" (UID: \"34ddeadc-2425-47fa-81d8-887f3488e6bf\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-4257c"
Sep 18 20:47:20.219564 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:47:20.219539    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wt48\" (UniqueName: \"kubernetes.io/projected/34ddeadc-2425-47fa-81d8-887f3488e6bf-kube-api-access-8wt48\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-4257c\" (UID: \"34ddeadc-2425-47fa-81d8-887f3488e6bf\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-4257c"
Sep 18 20:47:20.301552 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:47:20.301459    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-4257c"
Sep 18 20:47:20.581291 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:47:20.581267    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-4257c"]
Sep 18 20:47:20.583588 ip-10-0-128-161 kubenswrapper[2564]: W0918 20:47:20.583560    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34ddeadc_2425_47fa_81d8_887f3488e6bf.slice/crio-10efd6e35f2abbec44d4e9eb53d90de808daea5debedd36a8a04c284a801ee60 WatchSource:0}: Error finding container 10efd6e35f2abbec44d4e9eb53d90de808daea5debedd36a8a04c284a801ee60: Status 404 returned error can't find the container with id 10efd6e35f2abbec44d4e9eb53d90de808daea5debedd36a8a04c284a801ee60
Sep 18 20:47:20.789374 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:47:20.789326    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-4257c" event={"ID":"34ddeadc-2425-47fa-81d8-887f3488e6bf","Type":"ContainerStarted","Data":"57d8cf2417414d11d90dbd16a69046f219e4d90a5b649f745921c4d43a4465a6"}
Sep 18 20:47:20.789842 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:47:20.789396    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-4257c" event={"ID":"34ddeadc-2425-47fa-81d8-887f3488e6bf","Type":"ContainerStarted","Data":"10efd6e35f2abbec44d4e9eb53d90de808daea5debedd36a8a04c284a801ee60"}
Sep 18 20:47:20.789842 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:47:20.789574    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-4257c"
Sep 18 20:47:20.808230 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:47:20.808172    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-4257c" podStartSLOduration=1.808155682 podStartE2EDuration="1.808155682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:47:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:47:20.806537052 +0000 UTC m=+4157.316893167" watchObservedRunningTime="2026-09-18 20:47:20.808155682 +0000 UTC m=+4157.318511772"
Sep 18 20:47:28.345231 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:47:28.345192    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:47:39.281760 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:47:39.281721    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:47:39.290388 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:47:39.290355    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48mdlcg"]
Sep 18 20:47:39.347612 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:47:39.347577    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:47:46.895722 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:47:46.895676    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:47:47.221131 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:47:47.221045    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48mdlcg" podUID="439322c3-d40b-4902-83e7-1a6eaf95e6c8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:47:51.796937 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:47:51.796900    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-4257c"
Sep 18 20:47:54.312419 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:47:54.312379    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48mdlcg" podUID="439322c3-d40b-4902-83e7-1a6eaf95e6c8" containerName="main" containerID="cri-o://f01eb149038f3a59906d28d3157d2d52de176a915cf77243ed71de46b8a753f0" gracePeriod=45
Sep 18 20:47:57.221479 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:47:57.221437    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48mdlcg" podUID="439322c3-d40b-4902-83e7-1a6eaf95e6c8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:48:04.271304 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:48:04.271271    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-58vl4_945a444e-9381-4e2f-b4c1-64be0ec88e6c/console-operator/2.log"
Sep 18 20:48:04.281714 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:48:04.281691    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-58vl4_945a444e-9381-4e2f-b4c1-64be0ec88e6c/console-operator/2.log"
Sep 18 20:48:07.220933 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:48:07.220893    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48mdlcg" podUID="439322c3-d40b-4902-83e7-1a6eaf95e6c8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:48:07.221356 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:48:07.221052    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48mdlcg"
Sep 18 20:48:17.221872 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:48:17.221824    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48mdlcg" podUID="439322c3-d40b-4902-83e7-1a6eaf95e6c8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:48:24.572702 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:48:24.572634    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48mdlcg"
Sep 18 20:48:24.667472 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:48:24.667437    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/439322c3-d40b-4902-83e7-1a6eaf95e6c8-tls-certs\") pod \"439322c3-d40b-4902-83e7-1a6eaf95e6c8\" (UID: \"439322c3-d40b-4902-83e7-1a6eaf95e6c8\") "
Sep 18 20:48:24.667653 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:48:24.667501    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dgkg5\" (UniqueName: \"kubernetes.io/projected/439322c3-d40b-4902-83e7-1a6eaf95e6c8-kube-api-access-dgkg5\") pod \"439322c3-d40b-4902-83e7-1a6eaf95e6c8\" (UID: \"439322c3-d40b-4902-83e7-1a6eaf95e6c8\") "
Sep 18 20:48:24.669777 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:48:24.669748    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/439322c3-d40b-4902-83e7-1a6eaf95e6c8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "439322c3-d40b-4902-83e7-1a6eaf95e6c8" (UID: "439322c3-d40b-4902-83e7-1a6eaf95e6c8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:48:24.669892 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:48:24.669804    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/439322c3-d40b-4902-83e7-1a6eaf95e6c8-kube-api-access-dgkg5" (OuterVolumeSpecName: "kube-api-access-dgkg5") pod "439322c3-d40b-4902-83e7-1a6eaf95e6c8" (UID: "439322c3-d40b-4902-83e7-1a6eaf95e6c8"). InnerVolumeSpecName "kube-api-access-dgkg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:48:24.768449 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:48:24.768410    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/439322c3-d40b-4902-83e7-1a6eaf95e6c8-tls-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:48:24.768449 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:48:24.768443    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgkg5\" (UniqueName: \"kubernetes.io/projected/439322c3-d40b-4902-83e7-1a6eaf95e6c8-kube-api-access-dgkg5\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:48:25.101178 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:48:25.101146    2564 generic.go:356] "Generic (PLEG): container finished" podID="439322c3-d40b-4902-83e7-1a6eaf95e6c8" containerID="f01eb149038f3a59906d28d3157d2d52de176a915cf77243ed71de46b8a753f0" exitCode=0
Sep 18 20:48:25.101378 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:48:25.101195    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48mdlcg"
Sep 18 20:48:25.101378 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:48:25.101229    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48mdlcg" event={"ID":"439322c3-d40b-4902-83e7-1a6eaf95e6c8","Type":"ContainerDied","Data":"f01eb149038f3a59906d28d3157d2d52de176a915cf77243ed71de46b8a753f0"}
Sep 18 20:48:25.101378 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:48:25.101292    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48mdlcg" event={"ID":"439322c3-d40b-4902-83e7-1a6eaf95e6c8","Type":"ContainerDied","Data":"439cdf8e3509f794cd6ee980bb9b7fcbb2f43b665053473d6943553f8cd876a6"}
Sep 18 20:48:25.101378 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:48:25.101321    2564 scope.go:117] "RemoveContainer" containerID="f01eb149038f3a59906d28d3157d2d52de176a915cf77243ed71de46b8a753f0"
Sep 18 20:48:25.110378 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:48:25.110250    2564 scope.go:117] "RemoveContainer" containerID="f01eb149038f3a59906d28d3157d2d52de176a915cf77243ed71de46b8a753f0"
Sep 18 20:48:25.110691 ip-10-0-128-161 kubenswrapper[2564]: E0918 20:48:25.110663    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f01eb149038f3a59906d28d3157d2d52de176a915cf77243ed71de46b8a753f0\": container with ID starting with f01eb149038f3a59906d28d3157d2d52de176a915cf77243ed71de46b8a753f0 not found: ID does not exist" containerID="f01eb149038f3a59906d28d3157d2d52de176a915cf77243ed71de46b8a753f0"
Sep 18 20:48:25.110793 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:48:25.110700    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f01eb149038f3a59906d28d3157d2d52de176a915cf77243ed71de46b8a753f0"} err="failed to get container status \"f01eb149038f3a59906d28d3157d2d52de176a915cf77243ed71de46b8a753f0\": rpc error: code = NotFound desc = could not find container \"f01eb149038f3a59906d28d3157d2d52de176a915cf77243ed71de46b8a753f0\": container with ID starting with f01eb149038f3a59906d28d3157d2d52de176a915cf77243ed71de46b8a753f0 not found: ID does not exist"
Sep 18 20:48:25.125649 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:48:25.125609    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48mdlcg"]
Sep 18 20:48:25.131560 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:48:25.131533    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48mdlcg"]
Sep 18 20:48:26.000763 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:48:26.000669    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="439322c3-d40b-4902-83e7-1a6eaf95e6c8" path="/var/lib/kubelet/pods/439322c3-d40b-4902-83e7-1a6eaf95e6c8/volumes"
Sep 18 20:48:29.512778 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:48:29.512740    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:48:29.611056 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:48:29.611022    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:48:29.763745 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:48:29.763657    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:48:29.859185 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:48:29.859152    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:48:33.473373 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:48:33.473331    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:48:36.106162 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:48:36.106118    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:48:41.621121 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:48:41.621082    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:48:41.723135 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:48:41.723095    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:48:45.185752 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:48:45.185702    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:48:45.521502 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:48:45.521418    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45jq4f5"]
Sep 18 20:48:45.521791 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:48:45.521778    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="439322c3-d40b-4902-83e7-1a6eaf95e6c8" containerName="main"
Sep 18 20:48:45.521835 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:48:45.521792    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="439322c3-d40b-4902-83e7-1a6eaf95e6c8" containerName="main"
Sep 18 20:48:45.521891 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:48:45.521880    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="439322c3-d40b-4902-83e7-1a6eaf95e6c8" containerName="main"
Sep 18 20:48:45.526394 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:48:45.526371    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45jq4f5"
Sep 18 20:48:45.538824 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:48:45.538797    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 18 20:48:45.539187 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:48:45.539161    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 18 20:48:45.539374 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:48:45.539352    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 18 20:48:45.539480 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:48:45.539224    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-epp-sa-dockercfg-kg79x\""
Sep 18 20:48:45.539553 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:48:45.539212    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-4lw4c\""
Sep 18 20:48:45.542350 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:48:45.542293    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45jq4f5"]
Sep 18 20:48:45.551387 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:48:45.551352    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd3d54e7-13bf-4127-b005-5426a798ee33-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45jq4f5\" (UID: \"dd3d54e7-13bf-4127-b005-5426a798ee33\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45jq4f5"
Sep 18 20:48:45.551532 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:48:45.551414    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kvsb\" (UniqueName: \"kubernetes.io/projected/dd3d54e7-13bf-4127-b005-5426a798ee33-kube-api-access-5kvsb\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45jq4f5\" (UID: \"dd3d54e7-13bf-4127-b005-5426a798ee33\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45jq4f5"
Sep 18 20:48:45.607838 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:48:45.607804    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:48:45.651846 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:48:45.651809    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kvsb\" (UniqueName: \"kubernetes.io/projected/dd3d54e7-13bf-4127-b005-5426a798ee33-kube-api-access-5kvsb\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45jq4f5\" (UID: \"dd3d54e7-13bf-4127-b005-5426a798ee33\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45jq4f5"
Sep 18 20:48:45.652044 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:48:45.651895    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd3d54e7-13bf-4127-b005-5426a798ee33-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45jq4f5\" (UID: \"dd3d54e7-13bf-4127-b005-5426a798ee33\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45jq4f5"
Sep 18 20:48:45.654528 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:48:45.654505    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd3d54e7-13bf-4127-b005-5426a798ee33-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45jq4f5\" (UID: \"dd3d54e7-13bf-4127-b005-5426a798ee33\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45jq4f5"
Sep 18 20:48:45.663466 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:48:45.663438    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kvsb\" (UniqueName: \"kubernetes.io/projected/dd3d54e7-13bf-4127-b005-5426a798ee33-kube-api-access-5kvsb\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45jq4f5\" (UID: \"dd3d54e7-13bf-4127-b005-5426a798ee33\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45jq4f5"
Sep 18 20:48:45.708763 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:48:45.708726    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:48:45.838011 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:48:45.837870    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45jq4f5"
Sep 18 20:48:46.112023 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:48:46.111917    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45jq4f5"]
Sep 18 20:48:46.115064 ip-10-0-128-161 kubenswrapper[2564]: W0918 20:48:46.115033    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd3d54e7_13bf_4127_b005_5426a798ee33.slice/crio-0ea43cdd0943b5ba29a5f51e037b0395ad62b22eea226c74dd63d2fd99861cdf WatchSource:0}: Error finding container 0ea43cdd0943b5ba29a5f51e037b0395ad62b22eea226c74dd63d2fd99861cdf: Status 404 returned error can't find the container with id 0ea43cdd0943b5ba29a5f51e037b0395ad62b22eea226c74dd63d2fd99861cdf
Sep 18 20:48:46.209332 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:48:46.209287    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45jq4f5" event={"ID":"dd3d54e7-13bf-4127-b005-5426a798ee33","Type":"ContainerStarted","Data":"01f723cf8f28d5710f50df3a6cc4145449badebf082db5382e29dd2e77361e0d"}
Sep 18 20:48:46.209715 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:48:46.209348    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45jq4f5" event={"ID":"dd3d54e7-13bf-4127-b005-5426a798ee33","Type":"ContainerStarted","Data":"0ea43cdd0943b5ba29a5f51e037b0395ad62b22eea226c74dd63d2fd99861cdf"}
Sep 18 20:48:46.209715 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:48:46.209431    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45jq4f5"
Sep 18 20:48:46.227003 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:48:46.226910    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45jq4f5" podStartSLOduration=1.2268861819999999 podStartE2EDuration="1.226886182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:48:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:48:46.225658468 +0000 UTC m=+4242.736014580" watchObservedRunningTime="2026-09-18 20:48:46.226886182 +0000 UTC m=+4242.737242274"
Sep 18 20:49:17.216961 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:49:17.216927    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45jq4f5"
Sep 18 20:49:17.350697 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:49:17.350658    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:49:39.023595 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:49:39.023559    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-4257c"]
Sep 18 20:49:39.033962 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:49:39.033927    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:49:39.132030 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:49:39.131962    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:49:41.795995 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:49:41.795954    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-4257c" podUID="34ddeadc-2425-47fa-81d8-887f3488e6bf" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:49:51.796747 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:49:51.796700    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-4257c" podUID="34ddeadc-2425-47fa-81d8-887f3488e6bf" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:49:54.045799 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:49:54.045745    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-4257c" podUID="34ddeadc-2425-47fa-81d8-887f3488e6bf" containerName="main" containerID="cri-o://57d8cf2417414d11d90dbd16a69046f219e4d90a5b649f745921c4d43a4465a6" gracePeriod=45
Sep 18 20:50:01.796350 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:50:01.796262    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-4257c" podUID="34ddeadc-2425-47fa-81d8-887f3488e6bf" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:50:01.796704 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:50:01.796394    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-4257c"
Sep 18 20:50:11.796497 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:50:11.796459    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-4257c" podUID="34ddeadc-2425-47fa-81d8-887f3488e6bf" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:50:21.796642 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:50:21.796603    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-4257c" podUID="34ddeadc-2425-47fa-81d8-887f3488e6bf" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:50:24.305133 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:50:24.305053    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-4257c"
Sep 18 20:50:24.425413 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:50:24.425365    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34ddeadc-2425-47fa-81d8-887f3488e6bf-tls-certs\") pod \"34ddeadc-2425-47fa-81d8-887f3488e6bf\" (UID: \"34ddeadc-2425-47fa-81d8-887f3488e6bf\") "
Sep 18 20:50:24.425627 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:50:24.425443    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8wt48\" (UniqueName: \"kubernetes.io/projected/34ddeadc-2425-47fa-81d8-887f3488e6bf-kube-api-access-8wt48\") pod \"34ddeadc-2425-47fa-81d8-887f3488e6bf\" (UID: \"34ddeadc-2425-47fa-81d8-887f3488e6bf\") "
Sep 18 20:50:24.427825 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:50:24.427790    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34ddeadc-2425-47fa-81d8-887f3488e6bf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "34ddeadc-2425-47fa-81d8-887f3488e6bf" (UID: "34ddeadc-2425-47fa-81d8-887f3488e6bf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:50:24.427947 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:50:24.427863    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34ddeadc-2425-47fa-81d8-887f3488e6bf-kube-api-access-8wt48" (OuterVolumeSpecName: "kube-api-access-8wt48") pod "34ddeadc-2425-47fa-81d8-887f3488e6bf" (UID: "34ddeadc-2425-47fa-81d8-887f3488e6bf"). InnerVolumeSpecName "kube-api-access-8wt48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:50:24.526490 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:50:24.526447    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34ddeadc-2425-47fa-81d8-887f3488e6bf-tls-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:50:24.526490 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:50:24.526481    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wt48\" (UniqueName: \"kubernetes.io/projected/34ddeadc-2425-47fa-81d8-887f3488e6bf-kube-api-access-8wt48\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:50:24.697654 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:50:24.697607    2564 generic.go:356] "Generic (PLEG): container finished" podID="34ddeadc-2425-47fa-81d8-887f3488e6bf" containerID="57d8cf2417414d11d90dbd16a69046f219e4d90a5b649f745921c4d43a4465a6" exitCode=0
Sep 18 20:50:24.697654 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:50:24.697650    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-4257c"
Sep 18 20:50:24.697912 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:50:24.697685    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-4257c" event={"ID":"34ddeadc-2425-47fa-81d8-887f3488e6bf","Type":"ContainerDied","Data":"57d8cf2417414d11d90dbd16a69046f219e4d90a5b649f745921c4d43a4465a6"}
Sep 18 20:50:24.697912 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:50:24.697741    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-4257c" event={"ID":"34ddeadc-2425-47fa-81d8-887f3488e6bf","Type":"ContainerDied","Data":"10efd6e35f2abbec44d4e9eb53d90de808daea5debedd36a8a04c284a801ee60"}
Sep 18 20:50:24.697912 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:50:24.697763    2564 scope.go:117] "RemoveContainer" containerID="57d8cf2417414d11d90dbd16a69046f219e4d90a5b649f745921c4d43a4465a6"
Sep 18 20:50:24.707038 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:50:24.707010    2564 scope.go:117] "RemoveContainer" containerID="57d8cf2417414d11d90dbd16a69046f219e4d90a5b649f745921c4d43a4465a6"
Sep 18 20:50:24.707379 ip-10-0-128-161 kubenswrapper[2564]: E0918 20:50:24.707356    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57d8cf2417414d11d90dbd16a69046f219e4d90a5b649f745921c4d43a4465a6\": container with ID starting with 57d8cf2417414d11d90dbd16a69046f219e4d90a5b649f745921c4d43a4465a6 not found: ID does not exist" containerID="57d8cf2417414d11d90dbd16a69046f219e4d90a5b649f745921c4d43a4465a6"
Sep 18 20:50:24.707439 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:50:24.707388    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57d8cf2417414d11d90dbd16a69046f219e4d90a5b649f745921c4d43a4465a6"} err="failed to get container status \"57d8cf2417414d11d90dbd16a69046f219e4d90a5b649f745921c4d43a4465a6\": rpc error: code = NotFound desc = could not find container \"57d8cf2417414d11d90dbd16a69046f219e4d90a5b649f745921c4d43a4465a6\": container with ID starting with 57d8cf2417414d11d90dbd16a69046f219e4d90a5b649f745921c4d43a4465a6 not found: ID does not exist"
Sep 18 20:50:24.718138 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:50:24.718108    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-4257c"]
Sep 18 20:50:24.721679 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:50:24.721635    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-4257c"]
Sep 18 20:50:25.999156 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:50:25.999111    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34ddeadc-2425-47fa-81d8-887f3488e6bf" path="/var/lib/kubelet/pods/34ddeadc-2425-47fa-81d8-887f3488e6bf/volumes"
Sep 18 20:50:44.004722 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:50:44.004682    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:50:44.094118 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:50:44.094082    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:50:50.137108 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:50:50.137071    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:50:55.953751 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:50:55.953711    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:50:56.056539 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:50:56.056502    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:51:07.090810 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:07.090774    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-zlb9x"]
Sep 18 20:51:07.093203 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:07.091168    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34ddeadc-2425-47fa-81d8-887f3488e6bf" containerName="main"
Sep 18 20:51:07.093203 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:07.091181    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="34ddeadc-2425-47fa-81d8-887f3488e6bf" containerName="main"
Sep 18 20:51:07.093203 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:07.091239    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="34ddeadc-2425-47fa-81d8-887f3488e6bf" containerName="main"
Sep 18 20:51:07.094083 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:07.094065    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-zlb9x"
Sep 18 20:51:07.096283 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:07.096261    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Sep 18 20:51:07.097053 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:07.097029    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Sep 18 20:51:07.097197 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:07.097120    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Sep 18 20:51:07.097197 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:07.097149    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-ff7kp\""
Sep 18 20:51:07.105189 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:07.105161    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-zlb9x"]
Sep 18 20:51:07.126496 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:07.126453    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:51:07.203388 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:07.203335    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3030851-3899-4599-a22e-ee943554ff62-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-zlb9x\" (UID: \"f3030851-3899-4599-a22e-ee943554ff62\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-zlb9x"
Sep 18 20:51:07.203388 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:07.203394    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3030851-3899-4599-a22e-ee943554ff62-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-zlb9x\" (UID: \"f3030851-3899-4599-a22e-ee943554ff62\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-zlb9x"
Sep 18 20:51:07.203675 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:07.203497    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzlzf\" (UniqueName: \"kubernetes.io/projected/f3030851-3899-4599-a22e-ee943554ff62-kube-api-access-bzlzf\") pod \"rolling-upgrade-test-kserve-74b79d7f77-zlb9x\" (UID: \"f3030851-3899-4599-a22e-ee943554ff62\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-zlb9x"
Sep 18 20:51:07.203675 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:07.203561    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3030851-3899-4599-a22e-ee943554ff62-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-zlb9x\" (UID: \"f3030851-3899-4599-a22e-ee943554ff62\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-zlb9x"
Sep 18 20:51:07.203675 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:07.203597    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3030851-3899-4599-a22e-ee943554ff62-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-zlb9x\" (UID: \"f3030851-3899-4599-a22e-ee943554ff62\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-zlb9x"
Sep 18 20:51:07.203675 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:07.203626    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3030851-3899-4599-a22e-ee943554ff62-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-zlb9x\" (UID: \"f3030851-3899-4599-a22e-ee943554ff62\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-zlb9x"
Sep 18 20:51:07.304373 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:07.304330    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3030851-3899-4599-a22e-ee943554ff62-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-zlb9x\" (UID: \"f3030851-3899-4599-a22e-ee943554ff62\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-zlb9x"
Sep 18 20:51:07.304582 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:07.304456    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3030851-3899-4599-a22e-ee943554ff62-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-zlb9x\" (UID: \"f3030851-3899-4599-a22e-ee943554ff62\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-zlb9x"
Sep 18 20:51:07.304582 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:07.304519    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzlzf\" (UniqueName: \"kubernetes.io/projected/f3030851-3899-4599-a22e-ee943554ff62-kube-api-access-bzlzf\") pod \"rolling-upgrade-test-kserve-74b79d7f77-zlb9x\" (UID: \"f3030851-3899-4599-a22e-ee943554ff62\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-zlb9x"
Sep 18 20:51:07.304582 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:07.304572    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3030851-3899-4599-a22e-ee943554ff62-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-zlb9x\" (UID: \"f3030851-3899-4599-a22e-ee943554ff62\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-zlb9x"
Sep 18 20:51:07.304777 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:07.304607    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3030851-3899-4599-a22e-ee943554ff62-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-zlb9x\" (UID: \"f3030851-3899-4599-a22e-ee943554ff62\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-zlb9x"
Sep 18 20:51:07.304777 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:07.304635    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3030851-3899-4599-a22e-ee943554ff62-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-zlb9x\" (UID: \"f3030851-3899-4599-a22e-ee943554ff62\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-zlb9x"
Sep 18 20:51:07.304882 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:07.304841    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3030851-3899-4599-a22e-ee943554ff62-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-zlb9x\" (UID: \"f3030851-3899-4599-a22e-ee943554ff62\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-zlb9x"
Sep 18 20:51:07.304942 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:07.304890    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3030851-3899-4599-a22e-ee943554ff62-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-zlb9x\" (UID: \"f3030851-3899-4599-a22e-ee943554ff62\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-zlb9x"
Sep 18 20:51:07.305102 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:07.305063    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3030851-3899-4599-a22e-ee943554ff62-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-zlb9x\" (UID: \"f3030851-3899-4599-a22e-ee943554ff62\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-zlb9x"
Sep 18 20:51:07.307236 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:07.307210    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3030851-3899-4599-a22e-ee943554ff62-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-zlb9x\" (UID: \"f3030851-3899-4599-a22e-ee943554ff62\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-zlb9x"
Sep 18 20:51:07.307807 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:07.307787    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3030851-3899-4599-a22e-ee943554ff62-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-zlb9x\" (UID: \"f3030851-3899-4599-a22e-ee943554ff62\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-zlb9x"
Sep 18 20:51:07.319313 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:07.319272    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzlzf\" (UniqueName: \"kubernetes.io/projected/f3030851-3899-4599-a22e-ee943554ff62-kube-api-access-bzlzf\") pod \"rolling-upgrade-test-kserve-74b79d7f77-zlb9x\" (UID: \"f3030851-3899-4599-a22e-ee943554ff62\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-zlb9x"
Sep 18 20:51:07.334645 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:07.334604    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:51:07.406778 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:07.406740    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-zlb9x"
Sep 18 20:51:07.683315 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:07.683284    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-zlb9x"]
Sep 18 20:51:07.685795 ip-10-0-128-161 kubenswrapper[2564]: W0918 20:51:07.685763    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3030851_3899_4599_a22e_ee943554ff62.slice/crio-18fc580531ff165e63d599ff11386bf984e2563fba4271a80a3dca37becd184a WatchSource:0}: Error finding container 18fc580531ff165e63d599ff11386bf984e2563fba4271a80a3dca37becd184a: Status 404 returned error can't find the container with id 18fc580531ff165e63d599ff11386bf984e2563fba4271a80a3dca37becd184a
Sep 18 20:51:07.932037 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:07.931968    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-zlb9x" event={"ID":"f3030851-3899-4599-a22e-ee943554ff62","Type":"ContainerStarted","Data":"51b44ec2e094b19618fbefe41de24619179461b1998890dd2d558deb578060ed"}
Sep 18 20:51:07.932225 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:07.932057    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-zlb9x" event={"ID":"f3030851-3899-4599-a22e-ee943554ff62","Type":"ContainerStarted","Data":"18fc580531ff165e63d599ff11386bf984e2563fba4271a80a3dca37becd184a"}
Sep 18 20:51:08.407076 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:08.407035    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-zlb9x"
Sep 18 20:51:17.407590 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:17.407547    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-zlb9x"
Sep 18 20:51:17.421973 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:17.421933    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-zlb9x"
Sep 18 20:51:17.442552 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:17.442488    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-zlb9x" podStartSLOduration=10.442467662 podStartE2EDuration="10.442467662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:51:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:51:07.951577969 +0000 UTC m=+4384.461934083" watchObservedRunningTime="2026-09-18 20:51:17.442467662 +0000 UTC m=+4393.952823753"
Sep 18 20:51:18.009258 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:18.009229    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-zlb9x"
Sep 18 20:51:19.622551 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:19.622509    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:51:21.329889 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:21.329851    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45jq4f5"]
Sep 18 20:51:21.364720 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:21.364687    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:51:21.431501 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:21.431462    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:51:27.217530 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:27.217437    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45jq4f5" podUID="dd3d54e7-13bf-4127-b005-5426a798ee33" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:51:36.352637 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:36.352577    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45jq4f5" podUID="dd3d54e7-13bf-4127-b005-5426a798ee33" containerName="main" containerID="cri-o://01f723cf8f28d5710f50df3a6cc4145449badebf082db5382e29dd2e77361e0d" gracePeriod=45
Sep 18 20:51:37.217313 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:37.217273    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45jq4f5" podUID="dd3d54e7-13bf-4127-b005-5426a798ee33" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:51:47.217117 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:47.217077    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45jq4f5" podUID="dd3d54e7-13bf-4127-b005-5426a798ee33" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:51:47.217537 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:47.217181    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45jq4f5"
Sep 18 20:51:52.805785 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:52.805742    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-zlb9x"]
Sep 18 20:51:52.816142 ip-10-0-128-161 kubenswrapper[2564]: E0918 20:51:52.816103    2564 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 20:51:52.816142 ip-10-0-128-161 kubenswrapper[2564]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 20:51:52.816142 ip-10-0-128-161 kubenswrapper[2564]:         , stdout: , stderr: , exit code -1
Sep 18 20:51:52.816142 ip-10-0-128-161 kubenswrapper[2564]:  > containerID="51b44ec2e094b19618fbefe41de24619179461b1998890dd2d558deb578060ed" cmd=["/bin/sleep","15"]
Sep 18 20:51:52.816445 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:52.816161    2564 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-zlb9x" message="" err=<
Sep 18 20:51:52.816445 ip-10-0-128-161 kubenswrapper[2564]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 20:51:52.816445 ip-10-0-128-161 kubenswrapper[2564]:         , stdout: , stderr: , exit code -1
Sep 18 20:51:52.816445 ip-10-0-128-161 kubenswrapper[2564]:  >
Sep 18 20:51:52.816445 ip-10-0-128-161 kubenswrapper[2564]: E0918 20:51:52.816176    2564 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 20:51:52.816445 ip-10-0-128-161 kubenswrapper[2564]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 20:51:52.816445 ip-10-0-128-161 kubenswrapper[2564]:         , stdout: , stderr: , exit code -1
Sep 18 20:51:52.816445 ip-10-0-128-161 kubenswrapper[2564]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-zlb9x" podUID="f3030851-3899-4599-a22e-ee943554ff62" containerName="main" containerID="cri-o://51b44ec2e094b19618fbefe41de24619179461b1998890dd2d558deb578060ed"
Sep 18 20:51:52.816445 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:52.816215    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-zlb9x" podUID="f3030851-3899-4599-a22e-ee943554ff62" containerName="main" containerID="cri-o://51b44ec2e094b19618fbefe41de24619179461b1998890dd2d558deb578060ed" gracePeriod=60
Sep 18 20:51:52.998343 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:52.998306    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-zlb9x" podUID="f3030851-3899-4599-a22e-ee943554ff62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": dial tcp 10.133.0.80:8000: connect: connection refused"
Sep 18 20:51:53.166370 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:53.166342    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-zlb9x"
Sep 18 20:51:53.174626 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:53.174592    2564 generic.go:356] "Generic (PLEG): container finished" podID="f3030851-3899-4599-a22e-ee943554ff62" containerID="51b44ec2e094b19618fbefe41de24619179461b1998890dd2d558deb578060ed" exitCode=0
Sep 18 20:51:53.174786 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:53.174633    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-zlb9x"
Sep 18 20:51:53.174786 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:53.174677    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-zlb9x" event={"ID":"f3030851-3899-4599-a22e-ee943554ff62","Type":"ContainerDied","Data":"51b44ec2e094b19618fbefe41de24619179461b1998890dd2d558deb578060ed"}
Sep 18 20:51:53.174786 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:53.174738    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-zlb9x" event={"ID":"f3030851-3899-4599-a22e-ee943554ff62","Type":"ContainerDied","Data":"18fc580531ff165e63d599ff11386bf984e2563fba4271a80a3dca37becd184a"}
Sep 18 20:51:53.174786 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:53.174761    2564 scope.go:117] "RemoveContainer" containerID="51b44ec2e094b19618fbefe41de24619179461b1998890dd2d558deb578060ed"
Sep 18 20:51:53.184541 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:53.184057    2564 scope.go:117] "RemoveContainer" containerID="51b44ec2e094b19618fbefe41de24619179461b1998890dd2d558deb578060ed"
Sep 18 20:51:53.184760 ip-10-0-128-161 kubenswrapper[2564]: E0918 20:51:53.184555    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51b44ec2e094b19618fbefe41de24619179461b1998890dd2d558deb578060ed\": container with ID starting with 51b44ec2e094b19618fbefe41de24619179461b1998890dd2d558deb578060ed not found: ID does not exist" containerID="51b44ec2e094b19618fbefe41de24619179461b1998890dd2d558deb578060ed"
Sep 18 20:51:53.184760 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:53.184589    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51b44ec2e094b19618fbefe41de24619179461b1998890dd2d558deb578060ed"} err="failed to get container status \"51b44ec2e094b19618fbefe41de24619179461b1998890dd2d558deb578060ed\": rpc error: code = NotFound desc = could not find container \"51b44ec2e094b19618fbefe41de24619179461b1998890dd2d558deb578060ed\": container with ID starting with 51b44ec2e094b19618fbefe41de24619179461b1998890dd2d558deb578060ed not found: ID does not exist"
Sep 18 20:51:53.331508 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:53.331467    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3030851-3899-4599-a22e-ee943554ff62-home\") pod \"f3030851-3899-4599-a22e-ee943554ff62\" (UID: \"f3030851-3899-4599-a22e-ee943554ff62\") "
Sep 18 20:51:53.331709 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:53.331519    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3030851-3899-4599-a22e-ee943554ff62-dshm\") pod \"f3030851-3899-4599-a22e-ee943554ff62\" (UID: \"f3030851-3899-4599-a22e-ee943554ff62\") "
Sep 18 20:51:53.331709 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:53.331541    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3030851-3899-4599-a22e-ee943554ff62-tls-certs\") pod \"f3030851-3899-4599-a22e-ee943554ff62\" (UID: \"f3030851-3899-4599-a22e-ee943554ff62\") "
Sep 18 20:51:53.331709 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:53.331583    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3030851-3899-4599-a22e-ee943554ff62-model-cache\") pod \"f3030851-3899-4599-a22e-ee943554ff62\" (UID: \"f3030851-3899-4599-a22e-ee943554ff62\") "
Sep 18 20:51:53.331709 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:53.331625    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bzlzf\" (UniqueName: \"kubernetes.io/projected/f3030851-3899-4599-a22e-ee943554ff62-kube-api-access-bzlzf\") pod \"f3030851-3899-4599-a22e-ee943554ff62\" (UID: \"f3030851-3899-4599-a22e-ee943554ff62\") "
Sep 18 20:51:53.331709 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:53.331659    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3030851-3899-4599-a22e-ee943554ff62-tmp-dir\") pod \"f3030851-3899-4599-a22e-ee943554ff62\" (UID: \"f3030851-3899-4599-a22e-ee943554ff62\") "
Sep 18 20:51:53.332010 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:53.331804    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3030851-3899-4599-a22e-ee943554ff62-home" (OuterVolumeSpecName: "home") pod "f3030851-3899-4599-a22e-ee943554ff62" (UID: "f3030851-3899-4599-a22e-ee943554ff62"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:51:53.332010 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:53.331913    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3030851-3899-4599-a22e-ee943554ff62-model-cache" (OuterVolumeSpecName: "model-cache") pod "f3030851-3899-4599-a22e-ee943554ff62" (UID: "f3030851-3899-4599-a22e-ee943554ff62"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:51:53.332010 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:53.332005    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3030851-3899-4599-a22e-ee943554ff62-home\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:51:53.332171 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:53.332054    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3030851-3899-4599-a22e-ee943554ff62-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f3030851-3899-4599-a22e-ee943554ff62" (UID: "f3030851-3899-4599-a22e-ee943554ff62"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:51:53.334162 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:53.334104    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3030851-3899-4599-a22e-ee943554ff62-kube-api-access-bzlzf" (OuterVolumeSpecName: "kube-api-access-bzlzf") pod "f3030851-3899-4599-a22e-ee943554ff62" (UID: "f3030851-3899-4599-a22e-ee943554ff62"). InnerVolumeSpecName "kube-api-access-bzlzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:51:53.334367 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:53.334342    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3030851-3899-4599-a22e-ee943554ff62-dshm" (OuterVolumeSpecName: "dshm") pod "f3030851-3899-4599-a22e-ee943554ff62" (UID: "f3030851-3899-4599-a22e-ee943554ff62"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:51:53.334367 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:53.334344    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3030851-3899-4599-a22e-ee943554ff62-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f3030851-3899-4599-a22e-ee943554ff62" (UID: "f3030851-3899-4599-a22e-ee943554ff62"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:51:53.432927 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:53.432864    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3030851-3899-4599-a22e-ee943554ff62-tmp-dir\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:51:53.432927 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:53.432916    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3030851-3899-4599-a22e-ee943554ff62-dshm\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:51:53.432927 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:53.432928    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3030851-3899-4599-a22e-ee943554ff62-tls-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:51:53.432927 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:53.432939    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3030851-3899-4599-a22e-ee943554ff62-model-cache\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:51:53.433239 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:53.432951    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzlzf\" (UniqueName: \"kubernetes.io/projected/f3030851-3899-4599-a22e-ee943554ff62-kube-api-access-bzlzf\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:51:53.497145 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:53.497108    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-zlb9x"]
Sep 18 20:51:53.500967 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:53.500934    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-zlb9x"]
Sep 18 20:51:53.999406 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:53.999374    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3030851-3899-4599-a22e-ee943554ff62" path="/var/lib/kubelet/pods/f3030851-3899-4599-a22e-ee943554ff62/volumes"
Sep 18 20:51:57.217632 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:51:57.217591    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45jq4f5" podUID="dd3d54e7-13bf-4127-b005-5426a798ee33" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:52:06.612373 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:52:06.612289    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45jq4f5"
Sep 18 20:52:06.635912 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:52:06.635880    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5kvsb\" (UniqueName: \"kubernetes.io/projected/dd3d54e7-13bf-4127-b005-5426a798ee33-kube-api-access-5kvsb\") pod \"dd3d54e7-13bf-4127-b005-5426a798ee33\" (UID: \"dd3d54e7-13bf-4127-b005-5426a798ee33\") "
Sep 18 20:52:06.635912 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:52:06.635922    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd3d54e7-13bf-4127-b005-5426a798ee33-tls-certs\") pod \"dd3d54e7-13bf-4127-b005-5426a798ee33\" (UID: \"dd3d54e7-13bf-4127-b005-5426a798ee33\") "
Sep 18 20:52:06.638360 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:52:06.638311    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd3d54e7-13bf-4127-b005-5426a798ee33-kube-api-access-5kvsb" (OuterVolumeSpecName: "kube-api-access-5kvsb") pod "dd3d54e7-13bf-4127-b005-5426a798ee33" (UID: "dd3d54e7-13bf-4127-b005-5426a798ee33"). InnerVolumeSpecName "kube-api-access-5kvsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:52:06.638591 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:52:06.638558    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd3d54e7-13bf-4127-b005-5426a798ee33-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dd3d54e7-13bf-4127-b005-5426a798ee33" (UID: "dd3d54e7-13bf-4127-b005-5426a798ee33"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:52:06.736780 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:52:06.736743    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kvsb\" (UniqueName: \"kubernetes.io/projected/dd3d54e7-13bf-4127-b005-5426a798ee33-kube-api-access-5kvsb\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:52:06.736780 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:52:06.736776    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd3d54e7-13bf-4127-b005-5426a798ee33-tls-certs\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
Sep 18 20:52:07.156853 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:52:07.156768    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:52:07.215183 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:52:07.215147    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:52:07.247922 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:52:07.247885    2564 generic.go:356] "Generic (PLEG): container finished" podID="dd3d54e7-13bf-4127-b005-5426a798ee33" containerID="01f723cf8f28d5710f50df3a6cc4145449badebf082db5382e29dd2e77361e0d" exitCode=0
Sep 18 20:52:07.248135 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:52:07.247943    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45jq4f5"
Sep 18 20:52:07.248135 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:52:07.248003    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45jq4f5" event={"ID":"dd3d54e7-13bf-4127-b005-5426a798ee33","Type":"ContainerDied","Data":"01f723cf8f28d5710f50df3a6cc4145449badebf082db5382e29dd2e77361e0d"}
Sep 18 20:52:07.248135 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:52:07.248072    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45jq4f5" event={"ID":"dd3d54e7-13bf-4127-b005-5426a798ee33","Type":"ContainerDied","Data":"0ea43cdd0943b5ba29a5f51e037b0395ad62b22eea226c74dd63d2fd99861cdf"}
Sep 18 20:52:07.248135 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:52:07.248102    2564 scope.go:117] "RemoveContainer" containerID="01f723cf8f28d5710f50df3a6cc4145449badebf082db5382e29dd2e77361e0d"
Sep 18 20:52:07.259783 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:52:07.259749    2564 scope.go:117] "RemoveContainer" containerID="01f723cf8f28d5710f50df3a6cc4145449badebf082db5382e29dd2e77361e0d"
Sep 18 20:52:07.260242 ip-10-0-128-161 kubenswrapper[2564]: E0918 20:52:07.260219    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01f723cf8f28d5710f50df3a6cc4145449badebf082db5382e29dd2e77361e0d\": container with ID starting with 01f723cf8f28d5710f50df3a6cc4145449badebf082db5382e29dd2e77361e0d not found: ID does not exist" containerID="01f723cf8f28d5710f50df3a6cc4145449badebf082db5382e29dd2e77361e0d"
Sep 18 20:52:07.260315 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:52:07.260256    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01f723cf8f28d5710f50df3a6cc4145449badebf082db5382e29dd2e77361e0d"} err="failed to get container status \"01f723cf8f28d5710f50df3a6cc4145449badebf082db5382e29dd2e77361e0d\": rpc error: code = NotFound desc = could not find container \"01f723cf8f28d5710f50df3a6cc4145449badebf082db5382e29dd2e77361e0d\": container with ID starting with 01f723cf8f28d5710f50df3a6cc4145449badebf082db5382e29dd2e77361e0d not found: ID does not exist"
Sep 18 20:52:07.271653 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:52:07.271620    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45jq4f5"]
Sep 18 20:52:07.275507 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:52:07.275473    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45jq4f5"]
Sep 18 20:52:08.000109 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:52:08.000075    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd3d54e7-13bf-4127-b005-5426a798ee33" path="/var/lib/kubelet/pods/dd3d54e7-13bf-4127-b005-5426a798ee33/volumes"
Sep 18 20:52:10.444526 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:52:10.444488    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:52:15.498684 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:52:15.498636    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:52:37.152614 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:52:37.152568    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:52:38.251139 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:52:38.251107    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:52:43.504807 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:52:43.504772    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:52:53.450715 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:52:53.450664    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:52:53.725278 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:52:53.725176    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:52:55.011498 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:52:55.011453    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:52:55.087744 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:52:55.087705    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:52:58.124969 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:52:58.124859    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:52:58.223575 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:52:58.223535    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:52:59.740010 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:52:59.739952    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:53:04.301156 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:04.301125    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-58vl4_945a444e-9381-4e2f-b4c1-64be0ec88e6c/console-operator/2.log"
Sep 18 20:53:04.311568 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:04.311540    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-58vl4_945a444e-9381-4e2f-b4c1-64be0ec88e6c/console-operator/2.log"
Sep 18 20:53:05.323333 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:05.323285    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:53:05.395541 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:05.395500    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:53:17.035722 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:17.035686    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-k6rr9_05a22ef8-c4cb-4d4e-a375-4eadb4213fc2/discovery/0.log"
Sep 18 20:53:17.069079 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:17.069041    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-ggzph_a1c0f546-0824-446c-8f25-ffed46184e1c/istio-proxy/0.log"
Sep 18 20:53:17.085529 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:17.085495    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-ggzph_a1c0f546-0824-446c-8f25-ffed46184e1c/istio-proxy/1.log"
Sep 18 20:53:17.125074 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:17.125043    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-756fcb9d6-47xpw_20027a0c-0095-4418-aaa4-d0a6f273ef85/router/0.log"
Sep 18 20:53:18.012657 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:18.012607    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-k6rr9_05a22ef8-c4cb-4d4e-a375-4eadb4213fc2/discovery/0.log"
Sep 18 20:53:18.052429 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:18.052393    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-ggzph_a1c0f546-0824-446c-8f25-ffed46184e1c/istio-proxy/0.log"
Sep 18 20:53:18.080128 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:18.080099    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-ggzph_a1c0f546-0824-446c-8f25-ffed46184e1c/istio-proxy/1.log"
Sep 18 20:53:18.116676 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:18.116645    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-756fcb9d6-47xpw_20027a0c-0095-4418-aaa4-d0a6f273ef85/router/0.log"
Sep 18 20:53:19.347807 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:19.347774    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-ksj7l_2ee27ad9-6661-4074-9507-5fc815d026ca/manager/0.log"
Sep 18 20:53:19.360389 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:19.360362    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-8bfn7_81c1bdbc-a919-4ce9-9c99-501c8250ea0e/manager/0.log"
Sep 18 20:53:19.614600 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:19.614512    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-8rb5h_107236fd-4036-4417-a437-f674c2402002/manager/0.log"
Sep 18 20:53:22.262178 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:22.262147    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:53:22.322290 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:22.322253    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:53:22.407446 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:22.407413    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:53:22.476664 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:22.476625    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:53:25.283179 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:25.283131    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6vdpn_2939ca32-81a7-4e9d-bdc6-ffca73820918/global-pull-secret-syncer/0.log"
Sep 18 20:53:25.378133 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:25.378102    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log"
Sep 18 20:53:25.508963 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:25.508921    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-161.ec2.internal_9eb8c7aafdc391f2f7ae0f04a98530ea/haproxy/0.log"
Sep 18 20:53:28.687791 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:28.687747    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:53:28.784273 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:28.784238    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:53:30.680710 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:30.680669    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-ksj7l_2ee27ad9-6661-4074-9507-5fc815d026ca/manager/0.log"
Sep 18 20:53:30.717700 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:30.717675    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-8bfn7_81c1bdbc-a919-4ce9-9c99-501c8250ea0e/manager/0.log"
Sep 18 20:53:31.016556 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:31.016469    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-8rb5h_107236fd-4036-4417-a437-f674c2402002/manager/0.log"
Sep 18 20:53:32.528254 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:32.528222    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-n86fq_2ee2e08b-c234-4675-83ba-a76063ea1285/kube-state-metrics/0.log"
Sep 18 20:53:32.549226 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:32.549197    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-n86fq_2ee2e08b-c234-4675-83ba-a76063ea1285/kube-rbac-proxy-main/0.log"
Sep 18 20:53:32.581749 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:32.581719    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-n86fq_2ee2e08b-c234-4675-83ba-a76063ea1285/kube-rbac-proxy-self/0.log"
Sep 18 20:53:32.623330 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:32.623302    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-565bdf6bdc-qbdl7_73718dc5-55c2-4493-af09-39dd4b81dce2/metrics-server/0.log"
Sep 18 20:53:32.698619 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:32.698587    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2ls7m_d5526936-3945-4732-bb5e-515d905fa913/node-exporter/0.log"
Sep 18 20:53:32.736391 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:32.736306    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2ls7m_d5526936-3945-4732-bb5e-515d905fa913/kube-rbac-proxy/0.log"
Sep 18 20:53:32.766482 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:32.766454    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2ls7m_d5526936-3945-4732-bb5e-515d905fa913/init-textfile/0.log"
Sep 18 20:53:32.978922 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:32.978889    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-mxhbj_25c36b4f-3ee8-48f3-83aa-b816203e1335/kube-rbac-proxy-main/0.log"
Sep 18 20:53:33.003272 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:33.003184    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-mxhbj_25c36b4f-3ee8-48f3-83aa-b816203e1335/kube-rbac-proxy-self/0.log"
Sep 18 20:53:33.030425 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:33.030397    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-mxhbj_25c36b4f-3ee8-48f3-83aa-b816203e1335/openshift-state-metrics/0.log"
Sep 18 20:53:33.265040 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:33.264925    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-zf82v_e9a41502-dda4-4946-8ef5-bf9c8e5035b9/prometheus-operator/0.log"
Sep 18 20:53:33.288924 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:33.288890    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-zf82v_e9a41502-dda4-4946-8ef5-bf9c8e5035b9/kube-rbac-proxy/0.log"
Sep 18 20:53:33.316341 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:33.316307    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5c6747dc7b-xw6cl_c0f6cc20-e321-47c5-a2b0-ea3d9d29cbe5/prometheus-operator-admission-webhook/0.log"
Sep 18 20:53:35.098822 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:35.098794    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-58vl4_945a444e-9381-4e2f-b4c1-64be0ec88e6c/console-operator/2.log"
Sep 18 20:53:35.107410 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:35.107378    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-58vl4_945a444e-9381-4e2f-b4c1-64be0ec88e6c/console-operator/3.log"
Sep 18 20:53:35.319362 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:35.319323    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kcrhg/perf-node-gather-daemonset-bvx7n"]
Sep 18 20:53:35.319734 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:35.319720    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3030851-3899-4599-a22e-ee943554ff62" containerName="main"
Sep 18 20:53:35.319781 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:35.319736    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3030851-3899-4599-a22e-ee943554ff62" containerName="main"
Sep 18 20:53:35.319781 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:35.319744    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd3d54e7-13bf-4127-b005-5426a798ee33" containerName="main"
Sep 18 20:53:35.319781 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:35.319750    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd3d54e7-13bf-4127-b005-5426a798ee33" containerName="main"
Sep 18 20:53:35.319880 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:35.319817    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd3d54e7-13bf-4127-b005-5426a798ee33" containerName="main"
Sep 18 20:53:35.319880 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:35.319827    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3030851-3899-4599-a22e-ee943554ff62" containerName="main"
Sep 18 20:53:35.322841 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:35.322822    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kcrhg/perf-node-gather-daemonset-bvx7n"
Sep 18 20:53:35.325130 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:35.325099    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kcrhg\"/\"kube-root-ca.crt\""
Sep 18 20:53:35.325301 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:35.325205    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kcrhg\"/\"default-dockercfg-2sg28\""
Sep 18 20:53:35.325754 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:35.325736    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kcrhg\"/\"openshift-service-ca.crt\""
Sep 18 20:53:35.333132 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:35.333107    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kcrhg/perf-node-gather-daemonset-bvx7n"]
Sep 18 20:53:35.405829 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:35.405788    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ac3f0bd4-08e8-4f21-8ddd-1a91ad19031d-lib-modules\") pod \"perf-node-gather-daemonset-bvx7n\" (UID: \"ac3f0bd4-08e8-4f21-8ddd-1a91ad19031d\") " pod="openshift-must-gather-kcrhg/perf-node-gather-daemonset-bvx7n"
Sep 18 20:53:35.405829 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:35.405830    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ac3f0bd4-08e8-4f21-8ddd-1a91ad19031d-proc\") pod \"perf-node-gather-daemonset-bvx7n\" (UID: \"ac3f0bd4-08e8-4f21-8ddd-1a91ad19031d\") " pod="openshift-must-gather-kcrhg/perf-node-gather-daemonset-bvx7n"
Sep 18 20:53:35.406118 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:35.405853    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc6j9\" (UniqueName: \"kubernetes.io/projected/ac3f0bd4-08e8-4f21-8ddd-1a91ad19031d-kube-api-access-bc6j9\") pod \"perf-node-gather-daemonset-bvx7n\" (UID: \"ac3f0bd4-08e8-4f21-8ddd-1a91ad19031d\") " pod="openshift-must-gather-kcrhg/perf-node-gather-daemonset-bvx7n"
Sep 18 20:53:35.406118 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:35.405888    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ac3f0bd4-08e8-4f21-8ddd-1a91ad19031d-podres\") pod \"perf-node-gather-daemonset-bvx7n\" (UID: \"ac3f0bd4-08e8-4f21-8ddd-1a91ad19031d\") " pod="openshift-must-gather-kcrhg/perf-node-gather-daemonset-bvx7n"
Sep 18 20:53:35.406118 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:35.405968    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac3f0bd4-08e8-4f21-8ddd-1a91ad19031d-sys\") pod \"perf-node-gather-daemonset-bvx7n\" (UID: \"ac3f0bd4-08e8-4f21-8ddd-1a91ad19031d\") " pod="openshift-must-gather-kcrhg/perf-node-gather-daemonset-bvx7n"
Sep 18 20:53:35.506896 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:35.506853    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac3f0bd4-08e8-4f21-8ddd-1a91ad19031d-sys\") pod \"perf-node-gather-daemonset-bvx7n\" (UID: \"ac3f0bd4-08e8-4f21-8ddd-1a91ad19031d\") " pod="openshift-must-gather-kcrhg/perf-node-gather-daemonset-bvx7n"
Sep 18 20:53:35.507124 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:35.506923    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ac3f0bd4-08e8-4f21-8ddd-1a91ad19031d-lib-modules\") pod \"perf-node-gather-daemonset-bvx7n\" (UID: \"ac3f0bd4-08e8-4f21-8ddd-1a91ad19031d\") " pod="openshift-must-gather-kcrhg/perf-node-gather-daemonset-bvx7n"
Sep 18 20:53:35.507124 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:35.506950    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ac3f0bd4-08e8-4f21-8ddd-1a91ad19031d-proc\") pod \"perf-node-gather-daemonset-bvx7n\" (UID: \"ac3f0bd4-08e8-4f21-8ddd-1a91ad19031d\") " pod="openshift-must-gather-kcrhg/perf-node-gather-daemonset-bvx7n"
Sep 18 20:53:35.507124 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:35.506966    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc6j9\" (UniqueName: \"kubernetes.io/projected/ac3f0bd4-08e8-4f21-8ddd-1a91ad19031d-kube-api-access-bc6j9\") pod \"perf-node-gather-daemonset-bvx7n\" (UID: \"ac3f0bd4-08e8-4f21-8ddd-1a91ad19031d\") " pod="openshift-must-gather-kcrhg/perf-node-gather-daemonset-bvx7n"
Sep 18 20:53:35.507124 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:35.507016    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac3f0bd4-08e8-4f21-8ddd-1a91ad19031d-sys\") pod \"perf-node-gather-daemonset-bvx7n\" (UID: \"ac3f0bd4-08e8-4f21-8ddd-1a91ad19031d\") " pod="openshift-must-gather-kcrhg/perf-node-gather-daemonset-bvx7n"
Sep 18 20:53:35.507124 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:35.507025    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ac3f0bd4-08e8-4f21-8ddd-1a91ad19031d-podres\") pod \"perf-node-gather-daemonset-bvx7n\" (UID: \"ac3f0bd4-08e8-4f21-8ddd-1a91ad19031d\") " pod="openshift-must-gather-kcrhg/perf-node-gather-daemonset-bvx7n"
Sep 18 20:53:35.507124 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:35.507082    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ac3f0bd4-08e8-4f21-8ddd-1a91ad19031d-proc\") pod \"perf-node-gather-daemonset-bvx7n\" (UID: \"ac3f0bd4-08e8-4f21-8ddd-1a91ad19031d\") " pod="openshift-must-gather-kcrhg/perf-node-gather-daemonset-bvx7n"
Sep 18 20:53:35.507370 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:35.507128    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ac3f0bd4-08e8-4f21-8ddd-1a91ad19031d-podres\") pod \"perf-node-gather-daemonset-bvx7n\" (UID: \"ac3f0bd4-08e8-4f21-8ddd-1a91ad19031d\") " pod="openshift-must-gather-kcrhg/perf-node-gather-daemonset-bvx7n"
Sep 18 20:53:35.507370 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:35.507132    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ac3f0bd4-08e8-4f21-8ddd-1a91ad19031d-lib-modules\") pod \"perf-node-gather-daemonset-bvx7n\" (UID: \"ac3f0bd4-08e8-4f21-8ddd-1a91ad19031d\") " pod="openshift-must-gather-kcrhg/perf-node-gather-daemonset-bvx7n"
Sep 18 20:53:35.515478 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:35.515449    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc6j9\" (UniqueName: \"kubernetes.io/projected/ac3f0bd4-08e8-4f21-8ddd-1a91ad19031d-kube-api-access-bc6j9\") pod \"perf-node-gather-daemonset-bvx7n\" (UID: \"ac3f0bd4-08e8-4f21-8ddd-1a91ad19031d\") " pod="openshift-must-gather-kcrhg/perf-node-gather-daemonset-bvx7n"
Sep 18 20:53:35.630035 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:35.629963    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6594f4c75-x728t_946a3d52-df38-4631-b5ce-a94283b4c9a4/download-server/0.log"
Sep 18 20:53:35.634313 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:35.634287    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kcrhg/perf-node-gather-daemonset-bvx7n"
Sep 18 20:53:35.904793 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:35.904764    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kcrhg/perf-node-gather-daemonset-bvx7n"]
Sep 18 20:53:35.907027 ip-10-0-128-161 kubenswrapper[2564]: W0918 20:53:35.906977    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podac3f0bd4_08e8_4f21_8ddd_1a91ad19031d.slice/crio-fa34ad9b3ee22ab74f4f5c3716029af28e055d182535df844b1c5afbb5bfe529 WatchSource:0}: Error finding container fa34ad9b3ee22ab74f4f5c3716029af28e055d182535df844b1c5afbb5bfe529: Status 404 returned error can't find the container with id fa34ad9b3ee22ab74f4f5c3716029af28e055d182535df844b1c5afbb5bfe529
Sep 18 20:53:36.670180 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:36.670138    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kcrhg/perf-node-gather-daemonset-bvx7n" event={"ID":"ac3f0bd4-08e8-4f21-8ddd-1a91ad19031d","Type":"ContainerStarted","Data":"40318d68e1592a5358757b2bd8a7defc0057fa8ebca3234b0b8359bad57be041"}
Sep 18 20:53:36.670592 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:36.670207    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kcrhg/perf-node-gather-daemonset-bvx7n" event={"ID":"ac3f0bd4-08e8-4f21-8ddd-1a91ad19031d","Type":"ContainerStarted","Data":"fa34ad9b3ee22ab74f4f5c3716029af28e055d182535df844b1c5afbb5bfe529"}
Sep 18 20:53:36.670592 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:36.670255    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-kcrhg/perf-node-gather-daemonset-bvx7n"
Sep 18 20:53:36.686540 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:36.686483    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kcrhg/perf-node-gather-daemonset-bvx7n" podStartSLOduration=1.68646873 podStartE2EDuration="1.68646873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:53:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:53:36.685231938 +0000 UTC m=+4533.195588047" watchObservedRunningTime="2026-09-18 20:53:36.68646873 +0000 UTC m=+4533.196824820"
Sep 18 20:53:37.258000 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:37.257946    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5krlr_8b6ee93c-e19d-44f2-9ba9-ae133edb7686/dns/0.log"
Sep 18 20:53:37.281125 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:37.281093    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5krlr_8b6ee93c-e19d-44f2-9ba9-ae133edb7686/kube-rbac-proxy/0.log"
Sep 18 20:53:37.399348 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:37.399314    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dwwf5_3f7e2771-a1bc-4ac9-9b67-1c7c9473d895/dns-node-resolver/0.log"
Sep 18 20:53:37.993464 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:37.993428    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-67dc7fd598-rjlqj_f280ba4e-c665-4351-8450-144134ea6c8b/registry/0.log"
Sep 18 20:53:38.045272 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:38.045239    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8hdsx_71862b1f-4124-47a1-8942-60acbbb5bdfb/node-ca/0.log"
Sep 18 20:53:39.231970 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:39.231915    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-k6rr9_05a22ef8-c4cb-4d4e-a375-4eadb4213fc2/discovery/0.log"
Sep 18 20:53:39.256934 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:39.256892    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-ggzph_a1c0f546-0824-446c-8f25-ffed46184e1c/istio-proxy/0.log"
Sep 18 20:53:39.271318 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:39.271269    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-667km_ecb576b0-eafa-4997-9690-36f2ac243783/konnectivity-agent/0.log" limit=1024
Sep 18 20:53:39.286143 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:39.286097    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-ggzph_a1c0f546-0824-446c-8f25-ffed46184e1c/istio-proxy/1.log"
Sep 18 20:53:39.307207 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:39.307182    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-756fcb9d6-47xpw_20027a0c-0095-4418-aaa4-d0a6f273ef85/router/0.log"
Sep 18 20:53:39.863117 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:39.863087    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-d8v8m_218f155a-deaf-4ea6-8234-74b776e05a7b/serve-healthcheck-canary/0.log"
Sep 18 20:53:40.455351 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:40.455322    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jfgj8_2c9bbaf8-3c60-402a-a13f-03d8e012c7c1/kube-rbac-proxy/0.log"
Sep 18 20:53:40.482115 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:40.482087    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jfgj8_2c9bbaf8-3c60-402a-a13f-03d8e012c7c1/exporter/0.log"
Sep 18 20:53:40.505466 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:40.505437    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jfgj8_2c9bbaf8-3c60-402a-a13f-03d8e012c7c1/extractor/0.log"
Sep 18 20:53:42.684941 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:42.684911    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-kcrhg/perf-node-gather-daemonset-bvx7n"
Sep 18 20:53:44.425137 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:44.425101    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-b9b5d54f8-vpxc9_ccc8fe2a-105c-45ef-8f34-4e4b9a369e4d/manager/0.log"
Sep 18 20:53:45.102745 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:45.102650    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-76bdf4b755-9n2gc_aa507a8c-3237-4c2c-8ce7-82dd6ca706b2/manager/0.log"
Sep 18 20:53:45.180474 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:45.180426    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-k6rr9_05a22ef8-c4cb-4d4e-a375-4eadb4213fc2/discovery/0.log"
Sep 18 20:53:45.200043 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:45.199930    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-ggzph_a1c0f546-0824-446c-8f25-ffed46184e1c/istio-proxy/0.log"
Sep 18 20:53:45.240675 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:45.240646    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-ggzph_a1c0f546-0824-446c-8f25-ffed46184e1c/istio-proxy/1.log"
Sep 18 20:53:45.260620 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:45.260579    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-756fcb9d6-47xpw_20027a0c-0095-4418-aaa4-d0a6f273ef85/router/0.log"
Sep 18 20:53:46.436555 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:46.436508    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-k6rr9_05a22ef8-c4cb-4d4e-a375-4eadb4213fc2/discovery/0.log"
Sep 18 20:53:46.457403 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:46.457371    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-ggzph_a1c0f546-0824-446c-8f25-ffed46184e1c/istio-proxy/0.log"
Sep 18 20:53:46.496001 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:46.495955    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-ggzph_a1c0f546-0824-446c-8f25-ffed46184e1c/istio-proxy/1.log"
Sep 18 20:53:46.513064 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:46.513032    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-756fcb9d6-47xpw_20027a0c-0095-4418-aaa4-d0a6f273ef85/router/0.log"
Sep 18 20:53:46.914669 ip-10-0-128-161 kubenswrapper[2564]: I0918 20:53:46.914630    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-xh6rg_881044d1-5a58-4322-8903-63bee4b44ddd/manager/0.log"