Sep 17 20:48:54.740132 ip-10-0-139-36 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 20:48:55.153897 ip-10-0-139-36 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 20:48:55.153897 ip-10-0-139-36 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 20:48:55.153897 ip-10-0-139-36 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 20:48:55.153897 ip-10-0-139-36 kubenswrapper[2575]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 20:48:55.153897 ip-10-0-139-36 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 20:48:55.154950 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.154865    2575 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 20:48:55.159904 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.159889    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 20:48:55.159904 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.159904    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 20:48:55.159967 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.159908    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 20:48:55.159967 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.159911    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 20:48:55.159967 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.159914    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 20:48:55.159967 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.159917    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 20:48:55.159967 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.159923    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 20:48:55.159967 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.159928    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 20:48:55.159967 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.159931    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 20:48:55.159967 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.159934    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 20:48:55.159967 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.159937    2575 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 20:48:55.159967 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.159940    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 20:48:55.159967 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.159947    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 20:48:55.159967 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.159950    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 20:48:55.159967 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.159953    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 20:48:55.159967 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.159956    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 20:48:55.159967 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.159958    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 20:48:55.159967 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.159961    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 20:48:55.159967 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.159964    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 20:48:55.159967 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.159967    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 20:48:55.159967 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.159970    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 20:48:55.159967 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.159973    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 20:48:55.160436 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.159976    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 20:48:55.160436 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.159979    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 20:48:55.160436 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.159983    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 20:48:55.160436 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.159985    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 20:48:55.160436 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.159988    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 20:48:55.160436 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.159991    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 20:48:55.160436 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.159993    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 20:48:55.160436 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.159997    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 20:48:55.160436 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160000    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 20:48:55.160436 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160004    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 20:48:55.160436 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160006    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 20:48:55.160436 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160009    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 20:48:55.160436 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160012    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 20:48:55.160436 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160015    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 20:48:55.160436 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160017    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 20:48:55.160436 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160020    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 20:48:55.160436 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160022    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 20:48:55.160436 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160025    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 20:48:55.160436 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160028    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 20:48:55.160905 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160030    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 20:48:55.160905 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160033    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 20:48:55.160905 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160035    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 20:48:55.160905 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160038    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 20:48:55.160905 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160040    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 20:48:55.160905 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160043    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 20:48:55.160905 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160045    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 20:48:55.160905 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160048    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 20:48:55.160905 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160050    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 20:48:55.160905 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160052    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 20:48:55.160905 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160055    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 20:48:55.160905 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160057    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 20:48:55.160905 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160060    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 20:48:55.160905 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160062    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 20:48:55.160905 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160066    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 20:48:55.160905 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160068    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 20:48:55.160905 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160071    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 20:48:55.160905 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160073    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 20:48:55.160905 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160076    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 20:48:55.160905 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160078    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 20:48:55.161403 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160081    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 20:48:55.161403 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160084    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 20:48:55.161403 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160086    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 20:48:55.161403 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160089    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 20:48:55.161403 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160091    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 20:48:55.161403 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160094    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 20:48:55.161403 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160096    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 20:48:55.161403 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160099    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 20:48:55.161403 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160101    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 20:48:55.161403 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160104    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 20:48:55.161403 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160107    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 20:48:55.161403 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160109    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 20:48:55.161403 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160112    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 20:48:55.161403 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160115    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 20:48:55.161403 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160117    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 20:48:55.161403 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160120    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 20:48:55.161403 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160122    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 20:48:55.161403 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160125    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 20:48:55.161403 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160127    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 20:48:55.161403 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160130    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 20:48:55.161902 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160133    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 20:48:55.161902 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160135    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 20:48:55.161902 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160137    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 20:48:55.161902 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160140    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 20:48:55.161902 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160142    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 20:48:55.161902 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160145    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 20:48:55.161902 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160149    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 20:48:55.161902 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160154    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 20:48:55.161902 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160157    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 20:48:55.161902 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160159    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 20:48:55.161902 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160162    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 20:48:55.161902 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160165    2575 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 20:48:55.161902 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160168    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 20:48:55.161902 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160170    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 20:48:55.161902 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160173    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 20:48:55.161902 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160175    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 20:48:55.161902 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160178    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 20:48:55.161902 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160180    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 20:48:55.161902 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160877    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 20:48:55.162395 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160884    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 20:48:55.162395 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160888    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 20:48:55.162395 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160891    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 20:48:55.162395 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160894    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 20:48:55.162395 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160898    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 20:48:55.162395 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160901    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 20:48:55.162395 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160904    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 20:48:55.162395 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160906    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 20:48:55.162395 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160909    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 20:48:55.162395 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160912    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 20:48:55.162395 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160915    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 20:48:55.162395 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160918    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 20:48:55.162395 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160921    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 20:48:55.162395 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160923    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 20:48:55.162395 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160926    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 20:48:55.162395 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160928    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 20:48:55.162395 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160931    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 20:48:55.162395 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160933    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 20:48:55.162395 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160936    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 20:48:55.162870 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160939    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 20:48:55.162870 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160942    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 20:48:55.162870 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160945    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 20:48:55.162870 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160947    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 20:48:55.162870 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160950    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 20:48:55.162870 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160953    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 20:48:55.162870 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160956    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 20:48:55.162870 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160959    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 20:48:55.162870 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160962    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 20:48:55.162870 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160965    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 20:48:55.162870 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160967    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 20:48:55.162870 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160970    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 20:48:55.162870 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160973    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 20:48:55.162870 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160975    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 20:48:55.162870 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160977    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 20:48:55.162870 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160980    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 20:48:55.162870 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160982    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 20:48:55.162870 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160985    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 20:48:55.162870 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160988    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 20:48:55.163344 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160990    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 20:48:55.163344 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160993    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 20:48:55.163344 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160995    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 20:48:55.163344 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.160998    2575 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 20:48:55.163344 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161000    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 20:48:55.163344 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161003    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 20:48:55.163344 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161005    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 20:48:55.163344 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161009    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 20:48:55.163344 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161011    2575 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 20:48:55.163344 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161015    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 20:48:55.163344 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161019    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 20:48:55.163344 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161023    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 20:48:55.163344 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161027    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 20:48:55.163344 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161030    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 20:48:55.163344 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161033    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 20:48:55.163344 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161036    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 20:48:55.163344 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161039    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 20:48:55.163344 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161042    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 20:48:55.163344 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161045    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 20:48:55.163810 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161048    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 20:48:55.163810 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161050    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 20:48:55.163810 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161053    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 20:48:55.163810 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161055    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 20:48:55.163810 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161058    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 20:48:55.163810 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161060    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 20:48:55.163810 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161063    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 20:48:55.163810 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161065    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 20:48:55.163810 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161068    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 20:48:55.163810 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161070    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 20:48:55.163810 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161073    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 20:48:55.163810 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161075    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 20:48:55.163810 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161078    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 20:48:55.163810 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161082    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 20:48:55.163810 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161084    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 20:48:55.163810 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161087    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 20:48:55.163810 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161089    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 20:48:55.163810 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161092    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 20:48:55.163810 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161094    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 20:48:55.163810 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161097    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 20:48:55.164311 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161100    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 20:48:55.164311 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161102    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 20:48:55.164311 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161104    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 20:48:55.164311 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161107    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 20:48:55.164311 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161110    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 20:48:55.164311 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161112    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 20:48:55.164311 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161115    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 20:48:55.164311 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161118    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 20:48:55.164311 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161120    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 20:48:55.164311 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161123    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 20:48:55.164311 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161125    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 20:48:55.164311 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161128    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 20:48:55.164311 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161131    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 20:48:55.164311 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161134    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 20:48:55.164311 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161136    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 20:48:55.164311 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161139    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 20:48:55.164311 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161142    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 20:48:55.164311 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161144    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 20:48:55.164311 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161146    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 20:48:55.164311 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161149    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 20:48:55.164800 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161152    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 20:48:55.164800 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161524    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 20:48:55.164800 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161529    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 20:48:55.164800 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161533    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 20:48:55.164800 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161536    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 20:48:55.164800 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161540    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 20:48:55.164800 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161543    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 20:48:55.164800 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161546    2575 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 20:48:55.164800 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161549    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 20:48:55.164800 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161551    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 20:48:55.164800 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161554    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 20:48:55.164800 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161556    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 20:48:55.164800 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161559    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 20:48:55.164800 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161562    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 20:48:55.164800 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161564    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 20:48:55.164800 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161567    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 20:48:55.164800 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161569    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 20:48:55.164800 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161572    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 20:48:55.164800 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161575    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 20:48:55.164800 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161578    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 20:48:55.165315 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161580    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 20:48:55.165315 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161583    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 20:48:55.165315 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161586    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 20:48:55.165315 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161590    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 20:48:55.165315 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161593    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 20:48:55.165315 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161596    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 20:48:55.165315 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161599    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 20:48:55.165315 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161601    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 20:48:55.165315 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161604    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 20:48:55.165315 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161606    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 20:48:55.165315 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161609    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 20:48:55.165315 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161611    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 20:48:55.165315 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161614    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 20:48:55.165315 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161617    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 20:48:55.165315 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161620    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 20:48:55.165315 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161622    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 20:48:55.165315 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161625    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 20:48:55.165315 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161627    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 20:48:55.165315 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161630    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 20:48:55.165315 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161633    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 20:48:55.165806 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161635    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 20:48:55.165806 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161638    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 20:48:55.165806 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161640    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 20:48:55.165806 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161643    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 20:48:55.165806 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161645    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 20:48:55.165806 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161648    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 20:48:55.165806 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161651    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 20:48:55.165806 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161653    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 20:48:55.165806 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161656    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 20:48:55.165806 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161660    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 20:48:55.165806 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161663    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 20:48:55.165806 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161666    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 20:48:55.165806 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161669    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 20:48:55.165806 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161674    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 20:48:55.165806 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161677    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 20:48:55.165806 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161680    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 20:48:55.165806 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161683    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 20:48:55.165806 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161685    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 20:48:55.165806 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161688    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 20:48:55.166284 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161691    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 20:48:55.166284 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161693    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 20:48:55.166284 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161696    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 20:48:55.166284 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161698    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 20:48:55.166284 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161701    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 20:48:55.166284 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161703    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 20:48:55.166284 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161707    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 20:48:55.166284 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161709    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 20:48:55.166284 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161712    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 20:48:55.166284 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161714    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 20:48:55.166284 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161717    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 20:48:55.166284 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161719    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 20:48:55.166284 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161722    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 20:48:55.166284 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161725    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 20:48:55.166284 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161727    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 20:48:55.166284 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161730    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 20:48:55.166284 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161732    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 20:48:55.166284 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161735    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 20:48:55.166284 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161739    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 20:48:55.166754 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161742    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 20:48:55.166754 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161745    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 20:48:55.166754 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161747    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 20:48:55.166754 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161750    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 20:48:55.166754 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161752    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 20:48:55.166754 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161755    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 20:48:55.166754 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161757    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 20:48:55.166754 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161764    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 20:48:55.166754 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161766    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 20:48:55.166754 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161770    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 20:48:55.166754 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161773    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 20:48:55.166754 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161777    2575 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 20:48:55.166754 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161779    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 20:48:55.166754 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161782    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 20:48:55.166754 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161785    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 20:48:55.166754 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161788    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 20:48:55.166754 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161791    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 20:48:55.166754 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161794    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 20:48:55.166754 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161796    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 20:48:55.167261 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161799    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 20:48:55.167261 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161802    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 20:48:55.167261 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.161804    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 20:48:55.167261 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.161893    2575 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 20:48:55.167261 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.161928    2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 20:48:55.167261 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.161934    2575 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 20:48:55.167261 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.161939    2575 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 20:48:55.167261 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.161944    2575 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 20:48:55.167261 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.161948    2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 20:48:55.167261 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.161952    2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 20:48:55.167261 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.161956    2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 20:48:55.167261 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.161959    2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 20:48:55.167261 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.161964    2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 20:48:55.167261 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.161967    2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 20:48:55.167261 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.161971    2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 20:48:55.167261 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.161974    2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 20:48:55.167261 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.161977    2575 flags.go:64] FLAG: --cgroup-root=""
Sep 17 20:48:55.167261 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.161980    2575 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 20:48:55.167261 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.161983    2575 flags.go:64] FLAG: --client-ca-file=""
Sep 17 20:48:55.167261 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.161985    2575 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 20:48:55.167261 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.161989    2575 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 20:48:55.167261 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.161994    2575 flags.go:64] FLAG: --cluster-domain=""
Sep 17 20:48:55.167261 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.161997    2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 20:48:55.167261 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.161999    2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 20:48:55.167858 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162003    2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 20:48:55.167858 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162006    2575 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 20:48:55.167858 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162010    2575 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 20:48:55.167858 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162013    2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 20:48:55.167858 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162016    2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 20:48:55.167858 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162019    2575 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 20:48:55.167858 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162023    2575 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 20:48:55.167858 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162026    2575 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 20:48:55.167858 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162029    2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 20:48:55.167858 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162032    2575 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 20:48:55.167858 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162034    2575 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 20:48:55.167858 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162038    2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 20:48:55.167858 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162041    2575 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 20:48:55.167858 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162044    2575 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 20:48:55.167858 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162047    2575 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 20:48:55.167858 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162050    2575 flags.go:64] FLAG: --enable-server="true"
Sep 17 20:48:55.167858 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162053    2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 20:48:55.167858 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162058    2575 flags.go:64] FLAG: --event-burst="100"
Sep 17 20:48:55.167858 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162061    2575 flags.go:64] FLAG: --event-qps="50"
Sep 17 20:48:55.167858 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162064    2575 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 20:48:55.167858 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162068    2575 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 20:48:55.167858 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162071    2575 flags.go:64] FLAG: --eviction-hard=""
Sep 17 20:48:55.167858 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162080    2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 20:48:55.167858 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162082    2575 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 20:48:55.168510 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162086    2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 20:48:55.168510 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162089    2575 flags.go:64] FLAG: --eviction-soft=""
Sep 17 20:48:55.168510 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162092    2575 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 20:48:55.168510 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162095    2575 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 20:48:55.168510 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162098    2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 20:48:55.168510 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162101    2575 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 20:48:55.168510 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162104    2575 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 20:48:55.168510 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162107    2575 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 20:48:55.168510 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162109    2575 flags.go:64] FLAG: --feature-gates=""
Sep 17 20:48:55.168510 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162113    2575 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 20:48:55.168510 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162116    2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 20:48:55.168510 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162119    2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 20:48:55.168510 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162122    2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 20:48:55.168510 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162125    2575 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 20:48:55.168510 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162128    2575 flags.go:64] FLAG: --help="false"
Sep 17 20:48:55.168510 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162131    2575 flags.go:64] FLAG: --hostname-override="ip-10-0-139-36.ec2.internal"
Sep 17 20:48:55.168510 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162134    2575 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 20:48:55.168510 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162137    2575 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 20:48:55.168510 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162140    2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 20:48:55.168510 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162144    2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 20:48:55.168510 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162147    2575 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 20:48:55.168510 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162150    2575 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 20:48:55.168510 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162152    2575 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 20:48:55.169110 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162155    2575 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 20:48:55.169110 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162158    2575 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 20:48:55.169110 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162162    2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 20:48:55.169110 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162165    2575 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 20:48:55.169110 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162168    2575 flags.go:64] FLAG: --kube-reserved=""
Sep 17 20:48:55.169110 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162172    2575 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 20:48:55.169110 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162175    2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 20:48:55.169110 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162178    2575 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 20:48:55.169110 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162181    2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 20:48:55.169110 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162184    2575 flags.go:64] FLAG: --lock-file=""
Sep 17 20:48:55.169110 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162187    2575 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 20:48:55.169110 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162190    2575 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 20:48:55.169110 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162193    2575 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 20:48:55.169110 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162198    2575 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 20:48:55.169110 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162201    2575 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 20:48:55.169110 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162203    2575 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 20:48:55.169110 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162206    2575 flags.go:64] FLAG: --logging-format="text"
Sep 17 20:48:55.169110 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162209    2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 20:48:55.169110 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162212    2575 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 20:48:55.169110 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162215    2575 flags.go:64] FLAG: --manifest-url=""
Sep 17 20:48:55.169110 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162218    2575 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 20:48:55.169110 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162222    2575 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 20:48:55.169110 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162225    2575 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 20:48:55.169110 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162229    2575 flags.go:64] FLAG: --max-pods="110"
Sep 17 20:48:55.169110 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162232    2575 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 20:48:55.169718 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162235    2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 20:48:55.169718 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162238    2575 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 20:48:55.169718 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162241    2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 20:48:55.169718 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162244    2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 20:48:55.169718 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162247    2575 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 20:48:55.169718 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162250    2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 20:48:55.169718 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162257    2575 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 20:48:55.169718 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162260    2575 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 20:48:55.169718 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162263    2575 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 20:48:55.169718 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162267    2575 flags.go:64] FLAG: --pod-cidr=""
Sep 17 20:48:55.169718 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162270    2575 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 20:48:55.169718 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162276    2575 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 20:48:55.169718 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162279    2575 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 20:48:55.169718 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162284    2575 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 20:48:55.169718 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162286    2575 flags.go:64] FLAG: --port="10250"
Sep 17 20:48:55.169718 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162290    2575 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 20:48:55.169718 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162293    2575 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e27c9320a8d6ca37"
Sep 17 20:48:55.169718 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162296    2575 flags.go:64] FLAG: --qos-reserved=""
Sep 17 20:48:55.169718 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162298    2575 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 20:48:55.169718 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162301    2575 flags.go:64] FLAG: --register-node="true"
Sep 17 20:48:55.169718 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162304    2575 flags.go:64] FLAG: --register-with-taints=""
Sep 17 20:48:55.169718 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162308    2575 flags.go:64] FLAG: --registry-burst="10"
Sep 17 20:48:55.169718 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162311    2575 flags.go:64] FLAG: --registry-qps="5"
Sep 17 20:48:55.169718 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162313    2575 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 20:48:55.170336 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162316    2575 flags.go:64] FLAG: --reserved-memory=""
Sep 17 20:48:55.170336 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162320    2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 20:48:55.170336 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162323    2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 20:48:55.170336 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162326    2575 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 20:48:55.170336 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162329    2575 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 20:48:55.170336 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162331    2575 flags.go:64] FLAG: --runonce="false"
Sep 17 20:48:55.170336 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162334    2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 20:48:55.170336 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162337    2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 20:48:55.170336 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162340    2575 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 20:48:55.170336 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162343    2575 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 20:48:55.170336 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162346    2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 20:48:55.170336 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162349    2575 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 20:48:55.170336 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162351    2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 20:48:55.170336 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162354    2575 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 20:48:55.170336 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162357    2575 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 20:48:55.170336 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162361    2575 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 20:48:55.170336 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162364    2575 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 20:48:55.170336 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162366    2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 20:48:55.170336 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162369    2575 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 20:48:55.170336 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162373    2575 flags.go:64] FLAG: --system-cgroups=""
Sep 17 20:48:55.170336 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162377    2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 20:48:55.170336 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162384    2575 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 20:48:55.170336 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162387    2575 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 20:48:55.170336 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162390    2575 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 20:48:55.170336 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162395    2575 flags.go:64] FLAG: --tls-min-version=""
Sep 17 20:48:55.170992 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162397    2575 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 20:48:55.170992 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162401    2575 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 20:48:55.170992 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162404    2575 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 20:48:55.170992 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162407    2575 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 20:48:55.170992 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162410    2575 flags.go:64] FLAG: --v="2"
Sep 17 20:48:55.170992 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162414    2575 flags.go:64] FLAG: --version="false"
Sep 17 20:48:55.170992 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162418    2575 flags.go:64] FLAG: --vmodule=""
Sep 17 20:48:55.170992 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162421    2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 20:48:55.170992 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.162424    2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 20:48:55.170992 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162535    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 20:48:55.170992 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162539    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 20:48:55.170992 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162542    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 20:48:55.170992 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162545    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 20:48:55.170992 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162548    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 20:48:55.170992 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162551    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 20:48:55.170992 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162554    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 20:48:55.170992 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162557    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 20:48:55.170992 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162560    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 20:48:55.170992 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162563    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 20:48:55.170992 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162565    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 20:48:55.170992 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162568    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 20:48:55.170992 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162570    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 20:48:55.171527 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162573    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 20:48:55.171527 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162576    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 20:48:55.171527 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162578    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 20:48:55.171527 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162581    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 20:48:55.171527 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162583    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 20:48:55.171527 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162588    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 20:48:55.171527 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162592    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 20:48:55.171527 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162595    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 20:48:55.171527 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162598    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 20:48:55.171527 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162600    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 20:48:55.171527 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162603    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 20:48:55.171527 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162606    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 20:48:55.171527 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162608    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 20:48:55.171527 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162611    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 20:48:55.171527 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162614    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 20:48:55.171527 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162616    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 20:48:55.171527 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162619    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 20:48:55.171527 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162621    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 20:48:55.171527 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162624    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 20:48:55.171527 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162626    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 20:48:55.172044 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162629    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 20:48:55.172044 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162631    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 20:48:55.172044 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162634    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 20:48:55.172044 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162638    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 20:48:55.172044 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162642    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 20:48:55.172044 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162645    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 20:48:55.172044 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162647    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 20:48:55.172044 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162650    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 20:48:55.172044 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162653    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 20:48:55.172044 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162656    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 20:48:55.172044 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162659    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 20:48:55.172044 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162661    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 20:48:55.172044 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162664    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 20:48:55.172044 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162667    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 20:48:55.172044 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162669    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 20:48:55.172044 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162672    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 20:48:55.172044 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162674    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 20:48:55.172044 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162678    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 20:48:55.172044 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162682    2575 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 20:48:55.172044 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162685    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 20:48:55.172523 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162688    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 20:48:55.172523 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162690    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 20:48:55.172523 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162693    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 20:48:55.172523 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162696    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 20:48:55.172523 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162698    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 20:48:55.172523 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162701    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 20:48:55.172523 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162704    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 20:48:55.172523 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162706    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 20:48:55.172523 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162709    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 20:48:55.172523 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162711    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 20:48:55.172523 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162714    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 20:48:55.172523 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162717    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 20:48:55.172523 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162719    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 20:48:55.172523 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162721    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 20:48:55.172523 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162724    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 20:48:55.172523 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162727    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 20:48:55.172523 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162729    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 20:48:55.172523 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162732    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 20:48:55.172523 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162736    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 20:48:55.173004 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162739    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 20:48:55.173004 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162741    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 20:48:55.173004 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162744    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 20:48:55.173004 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162746    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 20:48:55.173004 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162749    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 20:48:55.173004 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162752    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 20:48:55.173004 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162754    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 20:48:55.173004 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162757    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 20:48:55.173004 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162760    2575 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 20:48:55.173004 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162762    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 20:48:55.173004 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162766    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 20:48:55.173004 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162770    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 20:48:55.173004 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162773    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 20:48:55.173004 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162776    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 20:48:55.173004 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162779    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 20:48:55.173004 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162781    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 20:48:55.173004 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162785    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 20:48:55.173004 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162788    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 20:48:55.173004 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162791    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 20:48:55.173004 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162793    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 20:48:55.173481 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162796    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 20:48:55.173481 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162799    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 20:48:55.173481 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162802    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 20:48:55.173481 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162804    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 20:48:55.173481 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162807    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 20:48:55.173481 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162810    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 20:48:55.173481 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.162812    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 20:48:55.173481 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.163729    2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 20:48:55.173481 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.171784    2575 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 20:48:55.173481 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.171801    2575 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 20:48:55.173481 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.171820    2575 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 20:48:55.173481 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.171826    2575 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 20:48:55.173481 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.171911    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 20:48:55.173481 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.171917    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 20:48:55.173824 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.171921    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 20:48:55.173824 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.171924    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 20:48:55.173824 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.171927    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 20:48:55.173824 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.171930    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 20:48:55.173824 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.171933    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 20:48:55.173824 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.171935    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 20:48:55.173824 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.171938    2575 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 20:48:55.173824 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.171941    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 20:48:55.173824 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.171944    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 20:48:55.173824 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.171946    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 20:48:55.173824 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.171949    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 20:48:55.173824 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.171951    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 20:48:55.173824 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.171954    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 20:48:55.173824 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.171957    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 20:48:55.173824 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.171960    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 20:48:55.173824 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.171962    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 20:48:55.173824 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.171965    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 20:48:55.173824 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.171968    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 20:48:55.173824 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.171970    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 20:48:55.173824 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.171973    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 20:48:55.174348 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.171975    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 20:48:55.174348 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.171978    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 20:48:55.174348 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.171980    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 20:48:55.174348 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.171984    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 20:48:55.174348 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.171988    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 20:48:55.174348 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.171990    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 20:48:55.174348 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.171993    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 20:48:55.174348 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172006    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 20:48:55.174348 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172009    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 20:48:55.174348 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172012    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 20:48:55.174348 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172015    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 20:48:55.174348 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172018    2575 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 20:48:55.174348 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172020    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 20:48:55.174348 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172023    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 20:48:55.174348 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172025    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 20:48:55.174348 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172028    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 20:48:55.174348 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172030    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 20:48:55.174348 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172033    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 20:48:55.174348 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172036    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 20:48:55.174816 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172038    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 20:48:55.174816 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172041    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 20:48:55.174816 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172043    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 20:48:55.174816 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172046    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 20:48:55.174816 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172049    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 20:48:55.174816 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172052    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 20:48:55.174816 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172054    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 20:48:55.174816 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172057    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 20:48:55.174816 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172060    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 20:48:55.174816 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172062    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 20:48:55.174816 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172065    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 20:48:55.174816 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172068    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 20:48:55.174816 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172070    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 20:48:55.174816 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172073    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 20:48:55.174816 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172076    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 20:48:55.174816 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172078    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 20:48:55.174816 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172081    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 20:48:55.174816 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172083    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 20:48:55.174816 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172086    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 20:48:55.174816 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172088    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 20:48:55.175330 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172091    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 20:48:55.175330 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172100    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 20:48:55.175330 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172102    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 20:48:55.175330 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172105    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 20:48:55.175330 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172108    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 20:48:55.175330 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172110    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 20:48:55.175330 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172113    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 20:48:55.175330 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172116    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 20:48:55.175330 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172118    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 20:48:55.175330 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172121    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 20:48:55.175330 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172124    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 20:48:55.175330 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172127    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 20:48:55.175330 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172129    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 20:48:55.175330 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172132    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 20:48:55.175330 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172134    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 20:48:55.175330 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172137    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 20:48:55.175330 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172139    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 20:48:55.175330 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172142    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 20:48:55.175330 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172144    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 20:48:55.175330 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172147    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 20:48:55.175817 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172149    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 20:48:55.175817 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172154    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 20:48:55.175817 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172158    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 20:48:55.175817 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172162    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 20:48:55.175817 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172166    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 20:48:55.175817 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172169    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 20:48:55.175817 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172172    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 20:48:55.175817 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172175    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 20:48:55.175817 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172178    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 20:48:55.175817 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172181    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 20:48:55.175817 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172184    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 20:48:55.175817 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172187    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 20:48:55.175817 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172190    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 20:48:55.175817 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172193    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 20:48:55.175817 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172201    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 20:48:55.175817 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172204    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 20:48:55.175817 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172207    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 20:48:55.175817 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172210    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 20:48:55.176268 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.172215    2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 20:48:55.176268 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172357    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 20:48:55.176268 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172362    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 20:48:55.176268 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172364    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 20:48:55.176268 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172367    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 20:48:55.176268 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172370    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 20:48:55.176268 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172373    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 20:48:55.176268 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172375    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 20:48:55.176268 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172378    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 20:48:55.176268 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172381    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 20:48:55.176268 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172383    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 20:48:55.176268 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172386    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 20:48:55.176268 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172388    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 20:48:55.176268 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172391    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 20:48:55.176268 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172393    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 20:48:55.176630 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172396    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 20:48:55.176630 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172399    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 20:48:55.176630 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172401    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 20:48:55.176630 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172404    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 20:48:55.176630 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172407    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 20:48:55.176630 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172409    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 20:48:55.176630 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172412    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 20:48:55.176630 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172415    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 20:48:55.176630 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172417    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 20:48:55.176630 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172420    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 20:48:55.176630 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172423    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 20:48:55.176630 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172426    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 20:48:55.176630 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172429    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 20:48:55.176630 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172431    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 20:48:55.176630 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172439    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 20:48:55.176630 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172444    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 20:48:55.176630 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172447    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 20:48:55.176630 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172450    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 20:48:55.176630 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172453    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 20:48:55.176630 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172456    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 20:48:55.177116 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172459    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 20:48:55.177116 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172461    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 20:48:55.177116 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172464    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 20:48:55.177116 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172466    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 20:48:55.177116 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172469    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 20:48:55.177116 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172471    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 20:48:55.177116 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172474    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 20:48:55.177116 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172476    2575 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 20:48:55.177116 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172479    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 20:48:55.177116 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172482    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 20:48:55.177116 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172484    2575 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 20:48:55.177116 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172487    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 20:48:55.177116 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172489    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 20:48:55.177116 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172492    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 20:48:55.177116 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172495    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 20:48:55.177116 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172497    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 20:48:55.177116 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172500    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 20:48:55.177116 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172502    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 20:48:55.177116 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172505    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 20:48:55.177116 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172507    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 20:48:55.177589 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172510    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 20:48:55.177589 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172512    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 20:48:55.177589 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172515    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 20:48:55.177589 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172518    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 20:48:55.177589 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172520    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 20:48:55.177589 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172523    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 20:48:55.177589 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172525    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 20:48:55.177589 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172533    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 20:48:55.177589 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172537    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 20:48:55.177589 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172539    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 20:48:55.177589 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172542    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 20:48:55.177589 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172545    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 20:48:55.177589 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172547    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 20:48:55.177589 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172550    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 20:48:55.177589 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172554    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 20:48:55.177589 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172557    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 20:48:55.177589 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172560    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 20:48:55.177589 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172564    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 20:48:55.177589 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172567    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 20:48:55.178189 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172570    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 20:48:55.178189 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172573    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 20:48:55.178189 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172576    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 20:48:55.178189 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172579    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 20:48:55.178189 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172581    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 20:48:55.178189 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172584    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 20:48:55.178189 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172587    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 20:48:55.178189 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172590    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 20:48:55.178189 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172592    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 20:48:55.178189 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172595    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 20:48:55.178189 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172598    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 20:48:55.178189 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172600    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 20:48:55.178189 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172603    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 20:48:55.178189 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172606    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 20:48:55.178189 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172609    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 20:48:55.178189 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172611    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 20:48:55.178189 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172614    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 20:48:55.178189 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172617    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 20:48:55.178189 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172619    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 20:48:55.178653 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172622    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 20:48:55.178653 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172624    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 20:48:55.178653 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172633    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 20:48:55.178653 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172635    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 20:48:55.178653 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172638    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 20:48:55.178653 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172640    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 20:48:55.178653 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:55.172643    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 20:48:55.178653 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.172648    2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 20:48:55.178653 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.172766    2575 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 20:48:55.178653 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.175265    2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 20:48:55.178653 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.176313    2575 server.go:1019] "Starting client certificate rotation"
Sep 17 20:48:55.178653 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.176411    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 20:48:55.178653 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.176446    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 20:48:55.179272 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.179257    2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 20:48:55.185576 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.185555    2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 20:48:55.202066 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.202046    2575 log.go:25] "Validated CRI v1 runtime API"
Sep 17 20:48:55.203843 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.203827    2575 log.go:25] "Validated CRI v1 image API"
Sep 17 20:48:55.204719 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.204703    2575 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 20:48:55.210417 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.210398    2575 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 ba982543-3e71-4f55-9b31-af5bd9dc0f22:/dev/nvme0n1p3 e73450aa-e9f5-4c34-9779-12450aefe08d:/dev/nvme0n1p4]
Sep 17 20:48:55.210477 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.210416    2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 20:48:55.211956 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.211938    2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 20:48:55.215755 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.215650    2575 manager.go:216] Machine: {Timestamp:2026-09-17 20:48:55.214228845 +0000 UTC m=+0.366673211 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3107042 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2227f325cbddf81b5a20dd2a3a80e7 SystemUUID:ec2227f3-25cb-ddf8-1b5a-20dd2a3a80e7 BootID:9e5c8029-9da5-4b8e-b40d-b3721705ac98 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ec:e1:4d:17 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ec:e1:4d:17 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:56:4b:86:bd:92:b2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 20:48:55.215755 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.215752    2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 20:48:55.215896 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.215865    2575 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 20:48:55.219329 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.219304    2575 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 20:48:55.219470 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.219331    2575 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-36.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 20:48:55.220892 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.220881    2575 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 20:48:55.220928 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.220894    2575 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 20:48:55.220928 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.220902    2575 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 20:48:55.221764 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.221754    2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 20:48:55.221834 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.221825    2575 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 20:48:55.227349 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.227337    2575 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 20:48:55.227454 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.227445    2575 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 20:48:55.227487 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.227473    2575 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 20:48:55.227487 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.227486    2575 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 20:48:55.227540 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.227501    2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 20:48:55.227540 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.227510    2575 kubelet.go:398] "Adding apiserver pod source"
Sep 17 20:48:55.227540 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.227523    2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 20:48:55.227869 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.227843    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 20:48:55.227939 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.227883    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 20:48:55.230342 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.230326    2575 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 20:48:55.234713 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.234694    2575 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 20:48:55.234788 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.234728    2575 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 20:48:55.236347 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.236334    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 20:48:55.236415 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.236350    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 20:48:55.236415 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.236357    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 20:48:55.236415 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.236362    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 20:48:55.236415 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.236371    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 20:48:55.236415 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.236376    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 20:48:55.236415 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.236391    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 20:48:55.236415 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.236396    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 20:48:55.236415 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.236402    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 20:48:55.236415 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.236408    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 20:48:55.236663 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.236423    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 20:48:55.236663 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.236428    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 20:48:55.237345 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.237333    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 20:48:55.237380 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.237348    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 20:48:55.238106 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:48:55.238087    2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 20:48:55.238157 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:48:55.238141    2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-36.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 20:48:55.241106 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.241091    2575 server.go:1268] "Started kubelet"
Sep 17 20:48:55.241200 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.241168    2575 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 20:48:55.241252 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.241205    2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 20:48:55.241298 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.241269    2575 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 20:48:55.241835 ip-10-0-139-36 systemd[1]: Started Kubernetes Kubelet.
Sep 17 20:48:55.242733 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.242716    2575 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 20:48:55.242950 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.242935    2575 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 20:48:55.248417 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:48:55.248396    2575 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 20:48:55.249720 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.249706    2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-36.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 20:48:55.250683 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:48:55.249551    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-36.ec2.internal.18d636c82953bf42  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-36.ec2.internal,UID:ip-10-0-139-36.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-36.ec2.internal,},FirstTimestamp:2026-09-17 20:48:55.241064258 +0000 UTC m=+0.393508624,LastTimestamp:2026-09-17 20:48:55.241064258 +0000 UTC m=+0.393508624,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-36.ec2.internal,}"
Sep 17 20:48:55.251790 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.251772    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 20:48:55.252493 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.252472    2575 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 20:48:55.252610 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:48:55.252590    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-36.ec2.internal\" not found"
Sep 17 20:48:55.253275 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.253254    2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 20:48:55.253359 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.253298    2575 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 20:48:55.253359 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.253311    2575 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 20:48:55.253442 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.253396    2575 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 20:48:55.253442 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.253406    2575 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 20:48:55.253505 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.253477    2575 factory.go:55] Registering systemd factory
Sep 17 20:48:55.253505 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.253502    2575 factory.go:223] Registration of the systemd container factory successfully
Sep 17 20:48:55.253721 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.253709    2575 factory.go:153] Registering CRI-O factory
Sep 17 20:48:55.253721 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.253717    2575 factory.go:223] Registration of the crio container factory successfully
Sep 17 20:48:55.253833 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.253757    2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 20:48:55.253833 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.253772    2575 factory.go:103] Registering Raw factory
Sep 17 20:48:55.253833 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.253793    2575 manager.go:1196] Started watching for new ooms in manager
Sep 17 20:48:55.254288 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.254276    2575 manager.go:349] Starting recovery of all containers
Sep 17 20:48:55.255898 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:48:55.255871    2575 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-36.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 20:48:55.256231 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:48:55.256057    2575 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 20:48:55.262305 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.262276    2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-knrbh"
Sep 17 20:48:55.269553 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.269536    2575 manager.go:354] Recovery completed
Sep 17 20:48:55.269917 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.269895    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-knrbh"
Sep 17 20:48:55.274933 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.274916    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 20:48:55.278875 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.278859    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-36.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 20:48:55.278942 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.278889    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-36.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 20:48:55.278942 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.278903    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-36.ec2.internal" event="NodeHasSufficientPID"
Sep 17 20:48:55.279490 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.279472    2575 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 20:48:55.279490 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.279489    2575 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 20:48:55.279608 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.279505    2575 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 20:48:55.279608 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.279476    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 20:48:55.280348 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.280331    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-36.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 20:48:55.280402 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.280359    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-36.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 20:48:55.280402 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.280370    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-36.ec2.internal" event="NodeHasSufficientPID"
Sep 17 20:48:55.281255 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:48:55.281179    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-36.ec2.internal.18d636c82b94aed8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-36.ec2.internal,UID:ip-10-0-139-36.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-36.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-36.ec2.internal,},FirstTimestamp:2026-09-17 20:48:55.278874328 +0000 UTC m=+0.431318696,LastTimestamp:2026-09-17 20:48:55.278874328 +0000 UTC m=+0.431318696,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-36.ec2.internal,}"
Sep 17 20:48:55.281556 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.281545    2575 policy_none.go:49] "None policy: Start"
Sep 17 20:48:55.281616 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.281560    2575 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 20:48:55.281616 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.281569    2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 20:48:55.283554 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.283543    2575 policy_none.go:47] "Start"
Sep 17 20:48:55.413786 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.317419    2575 manager.go:337] "Starting Device Plugin manager"
Sep 17 20:48:55.413786 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:48:55.317458    2575 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 20:48:55.413786 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.317472    2575 server.go:87] "Starting device plugin registration server"
Sep 17 20:48:55.413786 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.317719    2575 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 20:48:55.413786 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.317731    2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 20:48:55.413786 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.317876    2575 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 20:48:55.413786 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.317968    2575 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 20:48:55.413786 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.317988    2575 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 20:48:55.413786 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:48:55.318495    2575 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 20:48:55.413786 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:48:55.318543    2575 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-36.ec2.internal\" not found"
Sep 17 20:48:55.413786 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.349033    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 20:48:55.413786 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.350222    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 20:48:55.413786 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.350280    2575 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 20:48:55.413786 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.350308    2575 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 20:48:55.413786 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:48:55.350351    2575 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 20:48:55.413786 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.352038    2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 20:48:55.418826 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.418806    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 20:48:55.419636 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.419617    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-36.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 20:48:55.419700 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.419655    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-36.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 20:48:55.419700 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.419668    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-36.ec2.internal" event="NodeHasSufficientPID"
Sep 17 20:48:55.419700 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.419692    2575 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-36.ec2.internal"
Sep 17 20:48:55.428334 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.428313    2575 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-36.ec2.internal"
Sep 17 20:48:55.428380 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:48:55.428344    2575 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-36.ec2.internal\": node \"ip-10-0-139-36.ec2.internal\" not found"
Sep 17 20:48:55.446105 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:48:55.446081    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-36.ec2.internal\" not found"
Sep 17 20:48:55.451108 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.451091    2575 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-36.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-36.ec2.internal"]
Sep 17 20:48:55.451168 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.451158    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 20:48:55.452599 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.452585    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-36.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 20:48:55.452652 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.452611    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-36.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 20:48:55.452652 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.452620    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-36.ec2.internal" event="NodeHasSufficientPID"
Sep 17 20:48:55.455102 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.455079    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 20:48:55.455237 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.455222    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-36.ec2.internal"
Sep 17 20:48:55.455279 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.455254    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 20:48:55.455858 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.455831    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-36.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 20:48:55.455858 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.455839    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-36.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 20:48:55.455994 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.455875    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-36.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 20:48:55.455994 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.455876    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-36.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 20:48:55.455994 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.455885    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-36.ec2.internal" event="NodeHasSufficientPID"
Sep 17 20:48:55.455994 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.455891    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-36.ec2.internal" event="NodeHasSufficientPID"
Sep 17 20:48:55.458168 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.458153    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-36.ec2.internal"
Sep 17 20:48:55.458236 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.458186    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 20:48:55.459107 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.459093    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-36.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 20:48:55.459166 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.459117    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-36.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 20:48:55.459166 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.459127    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-36.ec2.internal" event="NodeHasSufficientPID"
Sep 17 20:48:55.486453 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:48:55.486430    2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-36.ec2.internal\" not found" node="ip-10-0-139-36.ec2.internal"
Sep 17 20:48:55.490832 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:48:55.490817    2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-36.ec2.internal\" not found" node="ip-10-0-139-36.ec2.internal"
Sep 17 20:48:55.546349 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:48:55.546325    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-36.ec2.internal\" not found"
Sep 17 20:48:55.554652 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.554627    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/92e7e66bfcc6d2b6f25c825e57da7ac2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-36.ec2.internal\" (UID: \"92e7e66bfcc6d2b6f25c825e57da7ac2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-36.ec2.internal"
Sep 17 20:48:55.554652 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.554654    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/92e7e66bfcc6d2b6f25c825e57da7ac2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-36.ec2.internal\" (UID: \"92e7e66bfcc6d2b6f25c825e57da7ac2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-36.ec2.internal"
Sep 17 20:48:55.554801 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.554673    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/21ea1fa7387769fb888cd296b1eefe37-config\") pod \"kube-apiserver-proxy-ip-10-0-139-36.ec2.internal\" (UID: \"21ea1fa7387769fb888cd296b1eefe37\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-36.ec2.internal"
Sep 17 20:48:55.646765 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:48:55.646738    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-36.ec2.internal\" not found"
Sep 17 20:48:55.655104 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.655082    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/21ea1fa7387769fb888cd296b1eefe37-config\") pod \"kube-apiserver-proxy-ip-10-0-139-36.ec2.internal\" (UID: \"21ea1fa7387769fb888cd296b1eefe37\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-36.ec2.internal"
Sep 17 20:48:55.655182 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.655122    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/21ea1fa7387769fb888cd296b1eefe37-config\") pod \"kube-apiserver-proxy-ip-10-0-139-36.ec2.internal\" (UID: \"21ea1fa7387769fb888cd296b1eefe37\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-36.ec2.internal"
Sep 17 20:48:55.655182 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.655161    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/92e7e66bfcc6d2b6f25c825e57da7ac2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-36.ec2.internal\" (UID: \"92e7e66bfcc6d2b6f25c825e57da7ac2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-36.ec2.internal"
Sep 17 20:48:55.655182 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.655178    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/92e7e66bfcc6d2b6f25c825e57da7ac2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-36.ec2.internal\" (UID: \"92e7e66bfcc6d2b6f25c825e57da7ac2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-36.ec2.internal"
Sep 17 20:48:55.655291 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.655203    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/92e7e66bfcc6d2b6f25c825e57da7ac2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-36.ec2.internal\" (UID: \"92e7e66bfcc6d2b6f25c825e57da7ac2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-36.ec2.internal"
Sep 17 20:48:55.655291 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.655231    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/92e7e66bfcc6d2b6f25c825e57da7ac2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-36.ec2.internal\" (UID: \"92e7e66bfcc6d2b6f25c825e57da7ac2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-36.ec2.internal"
Sep 17 20:48:55.747498 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:48:55.747463    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-36.ec2.internal\" not found"
Sep 17 20:48:55.790946 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.790914    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-36.ec2.internal"
Sep 17 20:48:55.795030 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:55.795006    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-36.ec2.internal"
Sep 17 20:48:55.848681 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:48:55.848521    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-36.ec2.internal\" not found"
Sep 17 20:48:55.949061 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:48:55.949030    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-36.ec2.internal\" not found"
Sep 17 20:48:56.049651 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:48:56.049586    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-36.ec2.internal\" not found"
Sep 17 20:48:56.150177 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:48:56.150148    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-36.ec2.internal\" not found"
Sep 17 20:48:56.176481 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:56.176452    2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 20:48:56.176853 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:56.176617    2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 20:48:56.250324 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:48:56.250291    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-36.ec2.internal\" not found"
Sep 17 20:48:56.277243 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:56.277206    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 20:43:55 +0000 UTC" deadline="2028-02-13 02:21:19.109918511 +0000 UTC"
Sep 17 20:48:56.277243 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:56.277238    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12317h32m22.832683669s"
Sep 17 20:48:56.350706 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:48:56.350683    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-36.ec2.internal\" not found"
Sep 17 20:48:56.354069 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:56.354049    2575 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 20:48:56.450771 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:56.450743    2575 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 20:48:56.450911 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:48:56.450832    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-36.ec2.internal\" not found"
Sep 17 20:48:56.499026 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:56.498987    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92e7e66bfcc6d2b6f25c825e57da7ac2.slice/crio-c0a6bfb6048116485cb275059e9bf374ac5e379ad90a6a8d2f037abffafb2383 WatchSource:0}: Error finding container c0a6bfb6048116485cb275059e9bf374ac5e379ad90a6a8d2f037abffafb2383: Status 404 returned error can't find the container with id c0a6bfb6048116485cb275059e9bf374ac5e379ad90a6a8d2f037abffafb2383
Sep 17 20:48:56.499284 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:56.499257    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21ea1fa7387769fb888cd296b1eefe37.slice/crio-d37d1bd8538c8139d153be836662179638f8dc9a20c9aed2193a91a397d7087b WatchSource:0}: Error finding container d37d1bd8538c8139d153be836662179638f8dc9a20c9aed2193a91a397d7087b: Status 404 returned error can't find the container with id d37d1bd8538c8139d153be836662179638f8dc9a20c9aed2193a91a397d7087b
Sep 17 20:48:56.504135 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:56.504118    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 20:48:56.552958 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:48:56.552927    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-36.ec2.internal\" not found"
Sep 17 20:48:56.660565 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:48:56.660493    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-36.ec2.internal\" not found"
Sep 17 20:48:56.760613 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:48:56.760577    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-36.ec2.internal\" not found"
Sep 17 20:48:56.768154 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:56.768135    2575 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 20:48:56.853821 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:56.853780    2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-36.ec2.internal"
Sep 17 20:48:56.870924 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:56.870901    2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 20:48:56.871887 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:56.871872    2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-36.ec2.internal"
Sep 17 20:48:56.878385 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:56.878362    2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 20:48:57.210342 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.210306    2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 20:48:57.228922 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.228899    2575 apiserver.go:52] "Watching apiserver"
Sep 17 20:48:57.236242 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.236219    2575 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 20:48:57.236655 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.236624    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cj2s8","openshift-cluster-node-tuning-operator/tuned-rk9ss","openshift-image-registry/node-ca-2lrgx","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-36.ec2.internal","openshift-network-diagnostics/network-check-target-l28wp","openshift-network-operator/iptables-alerter-c4h4p","openshift-ovn-kubernetes/ovnkube-node-tmg6m","kube-system/global-pull-secret-syncer-dpwhq","kube-system/konnectivity-agent-mk66j","openshift-dns/node-resolver-48p9f","openshift-multus/multus-additional-cni-plugins-zcsbp","openshift-multus/multus-r58z7","openshift-multus/network-metrics-daemon-8b6dt","kube-system/kube-apiserver-proxy-ip-10-0-139-36.ec2.internal"]
Sep 17 20:48:57.239435 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.239412    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-r58z7"
Sep 17 20:48:57.241842 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.241822    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rk9ss"
Sep 17 20:48:57.242620 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.242595    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 20:48:57.242620 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.242606    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-5xqt6\""
Sep 17 20:48:57.242898 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.242864    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 20:48:57.243012 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.242989    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 20:48:57.243140 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.243037    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 20:48:57.244035 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.244013    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 20:48:57.244156 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.244137    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-c8k8s\""
Sep 17 20:48:57.244354 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.244020    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 20:48:57.244513 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.244495    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8b6dt"
Sep 17 20:48:57.244599 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:48:57.244565    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8b6dt" podUID="d42a790e-18b7-4478-b0c7-3c7cc4372fff"
Sep 17 20:48:57.248631 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.248600    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-48p9f"
Sep 17 20:48:57.251680 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.251658    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 20:48:57.251817 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.251733    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7jxcv\""
Sep 17 20:48:57.252005 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.251976    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 20:48:57.252005 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.251985    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l28wp"
Sep 17 20:48:57.252143 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:48:57.252065    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l28wp" podUID="53dee621-a806-480a-80c6-ff174deb8ed4"
Sep 17 20:48:57.252709 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.252692    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 20:48:57.254362 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.254342    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-c4h4p"
Sep 17 20:48:57.257055 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.256790    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 20:48:57.257055 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.257011    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 20:48:57.257055 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.257054    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-72xgz\""
Sep 17 20:48:57.257253 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.257165    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 20:48:57.258030 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.257379    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m"
Sep 17 20:48:57.259580 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.259412    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dpwhq"
Sep 17 20:48:57.259580 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:48:57.259484    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dpwhq" podUID="334a13ac-ccb2-43eb-bd43-566981c058ff"
Sep 17 20:48:57.260049 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.260034    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5qh6s\""
Sep 17 20:48:57.260310 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.260294    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 20:48:57.260531 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.260513    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 20:48:57.261650 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.261630    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 20:48:57.261734 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.261652    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 20:48:57.261948 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.261896    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 20:48:57.262141 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.262120    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 20:48:57.262429 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.262412    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mk66j"
Sep 17 20:48:57.263708 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.263684    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/efccf4d3-5793-4300-ac5f-ea35e453da2f-etc-sysctl-conf\") pod \"tuned-rk9ss\" (UID: \"efccf4d3-5793-4300-ac5f-ea35e453da2f\") " pod="openshift-cluster-node-tuning-operator/tuned-rk9ss"
Sep 17 20:48:57.263817 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.263719    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/efccf4d3-5793-4300-ac5f-ea35e453da2f-etc-tuned\") pod \"tuned-rk9ss\" (UID: \"efccf4d3-5793-4300-ac5f-ea35e453da2f\") " pod="openshift-cluster-node-tuning-operator/tuned-rk9ss"
Sep 17 20:48:57.263817 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.263747    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd-cnibin\") pod \"multus-r58z7\" (UID: \"a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd\") " pod="openshift-multus/multus-r58z7"
Sep 17 20:48:57.263817 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.263770    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd-host-var-lib-cni-bin\") pod \"multus-r58z7\" (UID: \"a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd\") " pod="openshift-multus/multus-r58z7"
Sep 17 20:48:57.263817 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.263803    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd-host-run-multus-certs\") pod \"multus-r58z7\" (UID: \"a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd\") " pod="openshift-multus/multus-r58z7"
Sep 17 20:48:57.264059 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.263827    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/efccf4d3-5793-4300-ac5f-ea35e453da2f-tmp\") pod \"tuned-rk9ss\" (UID: \"efccf4d3-5793-4300-ac5f-ea35e453da2f\") " pod="openshift-cluster-node-tuning-operator/tuned-rk9ss"
Sep 17 20:48:57.264059 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.263868    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/77f9970b-70e4-446e-954f-a37db50b183d-host-slash\") pod \"iptables-alerter-c4h4p\" (UID: \"77f9970b-70e4-446e-954f-a37db50b183d\") " pod="openshift-network-operator/iptables-alerter-c4h4p"
Sep 17 20:48:57.264059 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.263912    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd-multus-daemon-config\") pod \"multus-r58z7\" (UID: \"a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd\") " pod="openshift-multus/multus-r58z7"
Sep 17 20:48:57.264059 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.263936    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd-etc-kubernetes\") pod \"multus-r58z7\" (UID: \"a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd\") " pod="openshift-multus/multus-r58z7"
Sep 17 20:48:57.264059 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.263999    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/efccf4d3-5793-4300-ac5f-ea35e453da2f-etc-kubernetes\") pod \"tuned-rk9ss\" (UID: \"efccf4d3-5793-4300-ac5f-ea35e453da2f\") " pod="openshift-cluster-node-tuning-operator/tuned-rk9ss"
Sep 17 20:48:57.264059 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.264026    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/efccf4d3-5793-4300-ac5f-ea35e453da2f-sys\") pod \"tuned-rk9ss\" (UID: \"efccf4d3-5793-4300-ac5f-ea35e453da2f\") " pod="openshift-cluster-node-tuning-operator/tuned-rk9ss"
Sep 17 20:48:57.264059 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.264051    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/efccf4d3-5793-4300-ac5f-ea35e453da2f-host\") pod \"tuned-rk9ss\" (UID: \"efccf4d3-5793-4300-ac5f-ea35e453da2f\") " pod="openshift-cluster-node-tuning-operator/tuned-rk9ss"
Sep 17 20:48:57.264374 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.264079    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrt5f\" (UniqueName: \"kubernetes.io/projected/efccf4d3-5793-4300-ac5f-ea35e453da2f-kube-api-access-hrt5f\") pod \"tuned-rk9ss\" (UID: \"efccf4d3-5793-4300-ac5f-ea35e453da2f\") " pod="openshift-cluster-node-tuning-operator/tuned-rk9ss"
Sep 17 20:48:57.264374 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.264105    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/77f9970b-70e4-446e-954f-a37db50b183d-iptables-alerter-script\") pod \"iptables-alerter-c4h4p\" (UID: \"77f9970b-70e4-446e-954f-a37db50b183d\") " pod="openshift-network-operator/iptables-alerter-c4h4p"
Sep 17 20:48:57.264374 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.264178    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd-host-var-lib-kubelet\") pod \"multus-r58z7\" (UID: \"a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd\") " pod="openshift-multus/multus-r58z7"
Sep 17 20:48:57.264374 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.264222    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd-multus-conf-dir\") pod \"multus-r58z7\" (UID: \"a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd\") " pod="openshift-multus/multus-r58z7"
Sep 17 20:48:57.264374 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.264250    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/efccf4d3-5793-4300-ac5f-ea35e453da2f-etc-modprobe-d\") pod \"tuned-rk9ss\" (UID: \"efccf4d3-5793-4300-ac5f-ea35e453da2f\") " pod="openshift-cluster-node-tuning-operator/tuned-rk9ss"
Sep 17 20:48:57.264374 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.264298    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/efccf4d3-5793-4300-ac5f-ea35e453da2f-lib-modules\") pod \"tuned-rk9ss\" (UID: \"efccf4d3-5793-4300-ac5f-ea35e453da2f\") " pod="openshift-cluster-node-tuning-operator/tuned-rk9ss"
Sep 17 20:48:57.264374 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.264354    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd-host-var-lib-cni-multus\") pod \"multus-r58z7\" (UID: \"a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd\") " pod="openshift-multus/multus-r58z7"
Sep 17 20:48:57.264665 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.264483    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/efccf4d3-5793-4300-ac5f-ea35e453da2f-etc-systemd\") pod \"tuned-rk9ss\" (UID: \"efccf4d3-5793-4300-ac5f-ea35e453da2f\") " pod="openshift-cluster-node-tuning-operator/tuned-rk9ss"
Sep 17 20:48:57.264665 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.264515    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9vzl\" (UniqueName: \"kubernetes.io/projected/d42a790e-18b7-4478-b0c7-3c7cc4372fff-kube-api-access-l9vzl\") pod \"network-metrics-daemon-8b6dt\" (UID: \"d42a790e-18b7-4478-b0c7-3c7cc4372fff\") " pod="openshift-multus/network-metrics-daemon-8b6dt"
Sep 17 20:48:57.264665 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.264544    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72wqn\" (UniqueName: \"kubernetes.io/projected/a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd-kube-api-access-72wqn\") pod \"multus-r58z7\" (UID: \"a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd\") " pod="openshift-multus/multus-r58z7"
Sep 17 20:48:57.264665 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.264602    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqcxv\" (UniqueName: \"kubernetes.io/projected/53dee621-a806-480a-80c6-ff174deb8ed4-kube-api-access-fqcxv\") pod \"network-check-target-l28wp\" (UID: \"53dee621-a806-480a-80c6-ff174deb8ed4\") " pod="openshift-network-diagnostics/network-check-target-l28wp"
Sep 17 20:48:57.264665 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.264642    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd-system-cni-dir\") pod \"multus-r58z7\" (UID: \"a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd\") " pod="openshift-multus/multus-r58z7"
Sep 17 20:48:57.264918 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.264673    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd-host-run-netns\") pod \"multus-r58z7\" (UID: \"a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd\") " pod="openshift-multus/multus-r58z7"
Sep 17 20:48:57.264918 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.264699    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd-hostroot\") pod \"multus-r58z7\" (UID: \"a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd\") " pod="openshift-multus/multus-r58z7"
Sep 17 20:48:57.264918 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.264755    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/efccf4d3-5793-4300-ac5f-ea35e453da2f-var-lib-kubelet\") pod \"tuned-rk9ss\" (UID: \"efccf4d3-5793-4300-ac5f-ea35e453da2f\") " pod="openshift-cluster-node-tuning-operator/tuned-rk9ss"
Sep 17 20:48:57.264918 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.264765    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cj2s8"
Sep 17 20:48:57.264918 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.264784    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d42a790e-18b7-4478-b0c7-3c7cc4372fff-metrics-certs\") pod \"network-metrics-daemon-8b6dt\" (UID: \"d42a790e-18b7-4478-b0c7-3c7cc4372fff\") " pod="openshift-multus/network-metrics-daemon-8b6dt"
Sep 17 20:48:57.264918 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.264849    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfqrm\" (UniqueName: \"kubernetes.io/projected/5af35da6-3bf6-49ca-9a78-0218c7166c83-kube-api-access-nfqrm\") pod \"node-resolver-48p9f\" (UID: \"5af35da6-3bf6-49ca-9a78-0218c7166c83\") " pod="openshift-dns/node-resolver-48p9f"
Sep 17 20:48:57.264918 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.264931    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4grcw\" (UniqueName: \"kubernetes.io/projected/77f9970b-70e4-446e-954f-a37db50b183d-kube-api-access-4grcw\") pod \"iptables-alerter-c4h4p\" (UID: \"77f9970b-70e4-446e-954f-a37db50b183d\") " pod="openshift-network-operator/iptables-alerter-c4h4p"
Sep 17 20:48:57.265293 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.264960    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd-cni-binary-copy\") pod \"multus-r58z7\" (UID: \"a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd\") " pod="openshift-multus/multus-r58z7"
Sep 17 20:48:57.265293 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.264992    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-wrntt\""
Sep 17 20:48:57.265293 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.264992    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd-multus-socket-dir-parent\") pod \"multus-r58z7\" (UID: \"a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd\") " pod="openshift-multus/multus-r58z7"
Sep 17 20:48:57.265293 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.265080    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd-host-run-k8s-cni-cncf-io\") pod \"multus-r58z7\" (UID: \"a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd\") " pod="openshift-multus/multus-r58z7"
Sep 17 20:48:57.265293 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.265111    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/efccf4d3-5793-4300-ac5f-ea35e453da2f-etc-sysconfig\") pod \"tuned-rk9ss\" (UID: \"efccf4d3-5793-4300-ac5f-ea35e453da2f\") " pod="openshift-cluster-node-tuning-operator/tuned-rk9ss"
Sep 17 20:48:57.265293 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.265119    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 20:48:57.265293 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.265133    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/efccf4d3-5793-4300-ac5f-ea35e453da2f-etc-sysctl-d\") pod \"tuned-rk9ss\" (UID: \"efccf4d3-5793-4300-ac5f-ea35e453da2f\") " pod="openshift-cluster-node-tuning-operator/tuned-rk9ss"
Sep 17 20:48:57.265293 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.265158    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/efccf4d3-5793-4300-ac5f-ea35e453da2f-run\") pod \"tuned-rk9ss\" (UID: \"efccf4d3-5793-4300-ac5f-ea35e453da2f\") " pod="openshift-cluster-node-tuning-operator/tuned-rk9ss"
Sep 17 20:48:57.265293 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.265202    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5af35da6-3bf6-49ca-9a78-0218c7166c83-hosts-file\") pod \"node-resolver-48p9f\" (UID: \"5af35da6-3bf6-49ca-9a78-0218c7166c83\") " pod="openshift-dns/node-resolver-48p9f"
Sep 17 20:48:57.265293 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.265226    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5af35da6-3bf6-49ca-9a78-0218c7166c83-tmp-dir\") pod \"node-resolver-48p9f\" (UID: \"5af35da6-3bf6-49ca-9a78-0218c7166c83\") " pod="openshift-dns/node-resolver-48p9f"
Sep 17 20:48:57.265293 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.265253    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd-multus-cni-dir\") pod \"multus-r58z7\" (UID: \"a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd\") " pod="openshift-multus/multus-r58z7"
Sep 17 20:48:57.265753 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.265383    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 20:48:57.267968 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.267633    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 20:48:57.268174 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.268018    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 20:48:57.268174 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.268113    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-m2snm\""
Sep 17 20:48:57.268549 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.268357    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 20:48:57.268645 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.268628    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zcsbp"
Sep 17 20:48:57.270011 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.269990    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2lrgx"
Sep 17 20:48:57.271171 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.270988    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-29hz2\""
Sep 17 20:48:57.271171 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.271059    2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 20:48:57.271683 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.271495    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 20:48:57.271820 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.271803    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 20:48:57.272369 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.272346    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 20:48:57.272723 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.272700    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 20:48:57.273529 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.272956    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 20:48:57.273529 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.273246    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lw42v\""
Sep 17 20:48:57.295101 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.294964    2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-4xdfl"
Sep 17 20:48:57.304096 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.304071    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-4xdfl"
Sep 17 20:48:57.354245 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.354212    2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 20:48:57.358571 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.358516    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-36.ec2.internal" event={"ID":"21ea1fa7387769fb888cd296b1eefe37","Type":"ContainerStarted","Data":"d37d1bd8538c8139d153be836662179638f8dc9a20c9aed2193a91a397d7087b"}
Sep 17 20:48:57.360195 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.360166    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-36.ec2.internal" event={"ID":"92e7e66bfcc6d2b6f25c825e57da7ac2","Type":"ContainerStarted","Data":"c0a6bfb6048116485cb275059e9bf374ac5e379ad90a6a8d2f037abffafb2383"}
Sep 17 20:48:57.365591 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.365560    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7d501871-a063-4d0a-9267-d4216673f3ef-host-run-netns\") pod \"ovnkube-node-tmg6m\" (UID: \"7d501871-a063-4d0a-9267-d4216673f3ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m"
Sep 17 20:48:57.365704 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.365603    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7d501871-a063-4d0a-9267-d4216673f3ef-node-log\") pod \"ovnkube-node-tmg6m\" (UID: \"7d501871-a063-4d0a-9267-d4216673f3ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m"
Sep 17 20:48:57.365704 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.365635    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd-multus-cni-dir\") pod \"multus-r58z7\" (UID: \"a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd\") " pod="openshift-multus/multus-r58z7"
Sep 17 20:48:57.365704 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.365662    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/efccf4d3-5793-4300-ac5f-ea35e453da2f-etc-sysctl-conf\") pod \"tuned-rk9ss\" (UID: \"efccf4d3-5793-4300-ac5f-ea35e453da2f\") " pod="openshift-cluster-node-tuning-operator/tuned-rk9ss"
Sep 17 20:48:57.365704 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.365688    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/efccf4d3-5793-4300-ac5f-ea35e453da2f-etc-tuned\") pod \"tuned-rk9ss\" (UID: \"efccf4d3-5793-4300-ac5f-ea35e453da2f\") " pod="openshift-cluster-node-tuning-operator/tuned-rk9ss"
Sep 17 20:48:57.365941 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.365714    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/334a13ac-ccb2-43eb-bd43-566981c058ff-kubelet-config\") pod \"global-pull-secret-syncer-dpwhq\" (UID: \"334a13ac-ccb2-43eb-bd43-566981c058ff\") " pod="kube-system/global-pull-secret-syncer-dpwhq"
Sep 17 20:48:57.365941 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.365743    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5ddbfff5-cd86-477e-9152-7b0969bb0054-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zcsbp\" (UID: \"5ddbfff5-cd86-477e-9152-7b0969bb0054\") " pod="openshift-multus/multus-additional-cni-plugins-zcsbp"
Sep 17 20:48:57.365941 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.365779    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d501871-a063-4d0a-9267-d4216673f3ef-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tmg6m\" (UID: \"7d501871-a063-4d0a-9267-d4216673f3ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m"
Sep 17 20:48:57.365941 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.365806    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7d501871-a063-4d0a-9267-d4216673f3ef-ovnkube-config\") pod \"ovnkube-node-tmg6m\" (UID: \"7d501871-a063-4d0a-9267-d4216673f3ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m"
Sep 17 20:48:57.365941 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.365831    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f3d70fb8-dd97-4d0e-99b1-694aa64834ca-registration-dir\") pod \"aws-ebs-csi-driver-node-cj2s8\" (UID: \"f3d70fb8-dd97-4d0e-99b1-694aa64834ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cj2s8"
Sep 17 20:48:57.365941 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.365857    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd-cnibin\") pod \"multus-r58z7\" (UID: \"a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd\") " pod="openshift-multus/multus-r58z7"
Sep 17 20:48:57.365941 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.365898    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd-host-var-lib-cni-bin\") pod \"multus-r58z7\" (UID: \"a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd\") " pod="openshift-multus/multus-r58z7"
Sep 17 20:48:57.365941 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.365925    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd-host-run-multus-certs\") pod \"multus-r58z7\" (UID: \"a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd\") " pod="openshift-multus/multus-r58z7"
Sep 17 20:48:57.366356 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.365949    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/efccf4d3-5793-4300-ac5f-ea35e453da2f-tmp\") pod \"tuned-rk9ss\" (UID: \"efccf4d3-5793-4300-ac5f-ea35e453da2f\") " pod="openshift-cluster-node-tuning-operator/tuned-rk9ss"
Sep 17 20:48:57.366356 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.365972    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/334a13ac-ccb2-43eb-bd43-566981c058ff-dbus\") pod \"global-pull-secret-syncer-dpwhq\" (UID: \"334a13ac-ccb2-43eb-bd43-566981c058ff\") " pod="kube-system/global-pull-secret-syncer-dpwhq"
Sep 17 20:48:57.366356 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.366004    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7d501871-a063-4d0a-9267-d4216673f3ef-run-systemd\") pod \"ovnkube-node-tmg6m\" (UID: \"7d501871-a063-4d0a-9267-d4216673f3ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m"
Sep 17 20:48:57.366356 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.366028    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7d501871-a063-4d0a-9267-d4216673f3ef-run-ovn\") pod \"ovnkube-node-tmg6m\" (UID: \"7d501871-a063-4d0a-9267-d4216673f3ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m"
Sep 17 20:48:57.366356 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.366055    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7d501871-a063-4d0a-9267-d4216673f3ef-log-socket\") pod \"ovnkube-node-tmg6m\" (UID: \"7d501871-a063-4d0a-9267-d4216673f3ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m"
Sep 17 20:48:57.366356 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.366081    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd-etc-kubernetes\") pod \"multus-r58z7\" (UID: \"a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd\") " pod="openshift-multus/multus-r58z7"
Sep 17 20:48:57.366356 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.366112    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/efccf4d3-5793-4300-ac5f-ea35e453da2f-etc-kubernetes\") pod \"tuned-rk9ss\" (UID: \"efccf4d3-5793-4300-ac5f-ea35e453da2f\") " pod="openshift-cluster-node-tuning-operator/tuned-rk9ss"
Sep 17 20:48:57.366356 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.366135    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/efccf4d3-5793-4300-ac5f-ea35e453da2f-host\") pod \"tuned-rk9ss\" (UID: \"efccf4d3-5793-4300-ac5f-ea35e453da2f\") " pod="openshift-cluster-node-tuning-operator/tuned-rk9ss"
Sep 17 20:48:57.366356 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.366160    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrt5f\" (UniqueName: \"kubernetes.io/projected/efccf4d3-5793-4300-ac5f-ea35e453da2f-kube-api-access-hrt5f\") pod \"tuned-rk9ss\" (UID: \"efccf4d3-5793-4300-ac5f-ea35e453da2f\") " pod="openshift-cluster-node-tuning-operator/tuned-rk9ss"
Sep 17 20:48:57.366356 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.366198    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7d501871-a063-4d0a-9267-d4216673f3ef-ovn-node-metrics-cert\") pod \"ovnkube-node-tmg6m\" (UID: \"7d501871-a063-4d0a-9267-d4216673f3ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m"
Sep 17 20:48:57.366356 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.366224    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f3d70fb8-dd97-4d0e-99b1-694aa64834ca-sys-fs\") pod \"aws-ebs-csi-driver-node-cj2s8\" (UID: \"f3d70fb8-dd97-4d0e-99b1-694aa64834ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cj2s8"
Sep 17 20:48:57.366356 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.366250    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0369efa0-b31a-4fa5-b426-95ba067ea731-host\") pod \"node-ca-2lrgx\" (UID: \"0369efa0-b31a-4fa5-b426-95ba067ea731\") " pod="openshift-image-registry/node-ca-2lrgx"
Sep 17 20:48:57.366356 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.366286    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ftff\" (UniqueName: \"kubernetes.io/projected/0369efa0-b31a-4fa5-b426-95ba067ea731-kube-api-access-4ftff\") pod \"node-ca-2lrgx\" (UID: \"0369efa0-b31a-4fa5-b426-95ba067ea731\") " pod="openshift-image-registry/node-ca-2lrgx"
Sep 17 20:48:57.366356 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.366320    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/efccf4d3-5793-4300-ac5f-ea35e453da2f-lib-modules\") pod \"tuned-rk9ss\" (UID: \"efccf4d3-5793-4300-ac5f-ea35e453da2f\") " pod="openshift-cluster-node-tuning-operator/tuned-rk9ss"
Sep 17 20:48:57.366356 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.366346    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt2tb\" (UniqueName: \"kubernetes.io/projected/5ddbfff5-cd86-477e-9152-7b0969bb0054-kube-api-access-lt2tb\") pod \"multus-additional-cni-plugins-zcsbp\" (UID: \"5ddbfff5-cd86-477e-9152-7b0969bb0054\") " pod="openshift-multus/multus-additional-cni-plugins-zcsbp"
Sep 17 20:48:57.367176 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.366371    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7d501871-a063-4d0a-9267-d4216673f3ef-systemd-units\") pod \"ovnkube-node-tmg6m\" (UID: \"7d501871-a063-4d0a-9267-d4216673f3ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m"
Sep 17 20:48:57.367176 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.366396    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7d501871-a063-4d0a-9267-d4216673f3ef-env-overrides\") pod \"ovnkube-node-tmg6m\" (UID: \"7d501871-a063-4d0a-9267-d4216673f3ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m"
Sep 17 20:48:57.367176 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.366438    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f3d70fb8-dd97-4d0e-99b1-694aa64834ca-device-dir\") pod \"aws-ebs-csi-driver-node-cj2s8\" (UID: \"f3d70fb8-dd97-4d0e-99b1-694aa64834ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cj2s8"
Sep 17 20:48:57.367176 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.366469    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9vzl\" (UniqueName: \"kubernetes.io/projected/d42a790e-18b7-4478-b0c7-3c7cc4372fff-kube-api-access-l9vzl\") pod \"network-metrics-daemon-8b6dt\" (UID: \"d42a790e-18b7-4478-b0c7-3c7cc4372fff\") " pod="openshift-multus/network-metrics-daemon-8b6dt"
Sep 17 20:48:57.367176 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.366494    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/334a13ac-ccb2-43eb-bd43-566981c058ff-original-pull-secret\") pod \"global-pull-secret-syncer-dpwhq\" (UID: \"334a13ac-ccb2-43eb-bd43-566981c058ff\") " pod="kube-system/global-pull-secret-syncer-dpwhq"
Sep 17 20:48:57.367176 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.366520    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7d501871-a063-4d0a-9267-d4216673f3ef-host-cni-bin\") pod \"ovnkube-node-tmg6m\" (UID: \"7d501871-a063-4d0a-9267-d4216673f3ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m"
Sep 17 20:48:57.367176 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.366547    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1a25b3d5-790d-41a8-bea7-4689843c7d71-agent-certs\") pod \"konnectivity-agent-mk66j\" (UID: \"1a25b3d5-790d-41a8-bea7-4689843c7d71\") " pod="kube-system/konnectivity-agent-mk66j"
Sep 17 20:48:57.367176 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.366572    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1a25b3d5-790d-41a8-bea7-4689843c7d71-konnectivity-ca\") pod \"konnectivity-agent-mk66j\" (UID: \"1a25b3d5-790d-41a8-bea7-4689843c7d71\") " pod="kube-system/konnectivity-agent-mk66j"
Sep 17 20:48:57.367176 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.366598    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5ddbfff5-cd86-477e-9152-7b0969bb0054-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zcsbp\" (UID: \"5ddbfff5-cd86-477e-9152-7b0969bb0054\") " pod="openshift-multus/multus-additional-cni-plugins-zcsbp"
Sep 17 20:48:57.367176 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.366623    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f3d70fb8-dd97-4d0e-99b1-694aa64834ca-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cj2s8\" (UID: \"f3d70fb8-dd97-4d0e-99b1-694aa64834ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cj2s8"
Sep 17 20:48:57.367176 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.366650    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd-system-cni-dir\") pod \"multus-r58z7\" (UID: \"a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd\") " pod="openshift-multus/multus-r58z7"
Sep 17 20:48:57.367176 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.366676    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd-host-run-netns\") pod \"multus-r58z7\" (UID: \"a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd\") " pod="openshift-multus/multus-r58z7"
Sep 17 20:48:57.367176 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.366769    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd-system-cni-dir\") pod \"multus-r58z7\" (UID: \"a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd\") " pod="openshift-multus/multus-r58z7"
Sep 17 20:48:57.367176 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.366780    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd-host-var-lib-cni-bin\") pod \"multus-r58z7\" (UID: \"a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd\") " pod="openshift-multus/multus-r58z7"
Sep 17 20:48:57.367176 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.366819    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd-host-run-multus-certs\") pod \"multus-r58z7\" (UID: \"a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd\") " pod="openshift-multus/multus-r58z7"
Sep 17 20:48:57.367176 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.366842    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/efccf4d3-5793-4300-ac5f-ea35e453da2f-etc-sysctl-conf\") pod \"tuned-rk9ss\" (UID: \"efccf4d3-5793-4300-ac5f-ea35e453da2f\") " pod="openshift-cluster-node-tuning-operator/tuned-rk9ss"
Sep 17 20:48:57.367176 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.366998    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd-hostroot\") pod \"multus-r58z7\" (UID: \"a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd\") " pod="openshift-multus/multus-r58z7"
Sep 17 20:48:57.368085 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.367000    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd-multus-cni-dir\") pod \"multus-r58z7\" (UID: \"a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd\") " pod="openshift-multus/multus-r58z7"
Sep 17 20:48:57.368085 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.367015    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/efccf4d3-5793-4300-ac5f-ea35e453da2f-lib-modules\") pod \"tuned-rk9ss\" (UID: \"efccf4d3-5793-4300-ac5f-ea35e453da2f\") " pod="openshift-cluster-node-tuning-operator/tuned-rk9ss"
Sep 17 20:48:57.368085 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.367042    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/efccf4d3-5793-4300-ac5f-ea35e453da2f-host\") pod \"tuned-rk9ss\" (UID: \"efccf4d3-5793-4300-ac5f-ea35e453da2f\") " pod="openshift-cluster-node-tuning-operator/tuned-rk9ss"
Sep 17 20:48:57.368085 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.367057    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/efccf4d3-5793-4300-ac5f-ea35e453da2f-var-lib-kubelet\") pod \"tuned-rk9ss\" (UID: \"efccf4d3-5793-4300-ac5f-ea35e453da2f\") " pod="openshift-cluster-node-tuning-operator/tuned-rk9ss"
Sep 17 20:48:57.368085 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.367098    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd-etc-kubernetes\") pod \"multus-r58z7\" (UID: \"a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd\") " pod="openshift-multus/multus-r58z7"
Sep 17 20:48:57.368085 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.367099    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd-host-run-netns\") pod \"multus-r58z7\" (UID: \"a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd\") " pod="openshift-multus/multus-r58z7"
Sep 17 20:48:57.368085 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.367128    2575 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 20:48:57.368085 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.367141    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/efccf4d3-5793-4300-ac5f-ea35e453da2f-etc-kubernetes\") pod \"tuned-rk9ss\" (UID: \"efccf4d3-5793-4300-ac5f-ea35e453da2f\") " pod="openshift-cluster-node-tuning-operator/tuned-rk9ss"
Sep 17 20:48:57.368085 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.367144    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd-cnibin\") pod \"multus-r58z7\" (UID: \"a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd\") " pod="openshift-multus/multus-r58z7"
Sep 17 20:48:57.368085 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.367163    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/efccf4d3-5793-4300-ac5f-ea35e453da2f-var-lib-kubelet\") pod \"tuned-rk9ss\" (UID: \"efccf4d3-5793-4300-ac5f-ea35e453da2f\") " pod="openshift-cluster-node-tuning-operator/tuned-rk9ss"
Sep 17 20:48:57.368085 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.367191    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd-hostroot\") pod \"multus-r58z7\" (UID: \"a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd\") " pod="openshift-multus/multus-r58z7"
Sep 17 20:48:57.368085 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.367194    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d42a790e-18b7-4478-b0c7-3c7cc4372fff-metrics-certs\") pod \"network-metrics-daemon-8b6dt\" (UID: \"d42a790e-18b7-4478-b0c7-3c7cc4372fff\") " pod="openshift-multus/network-metrics-daemon-8b6dt"
Sep 17 20:48:57.368085 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:48:57.367266    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:48:57.368085 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.367303    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4grcw\" (UniqueName: \"kubernetes.io/projected/77f9970b-70e4-446e-954f-a37db50b183d-kube-api-access-4grcw\") pod \"iptables-alerter-c4h4p\" (UID: \"77f9970b-70e4-446e-954f-a37db50b183d\") " pod="openshift-network-operator/iptables-alerter-c4h4p"
Sep 17 20:48:57.368085 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:48:57.367351    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d42a790e-18b7-4478-b0c7-3c7cc4372fff-metrics-certs podName:d42a790e-18b7-4478-b0c7-3c7cc4372fff nodeName:}" failed. No retries permitted until 2026-09-17 20:48:57.867327009 +0000 UTC m=+3.019771381 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d42a790e-18b7-4478-b0c7-3c7cc4372fff-metrics-certs") pod "network-metrics-daemon-8b6dt" (UID: "d42a790e-18b7-4478-b0c7-3c7cc4372fff") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:48:57.368085 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.367373    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7d501871-a063-4d0a-9267-d4216673f3ef-host-cni-netd\") pod \"ovnkube-node-tmg6m\" (UID: \"7d501871-a063-4d0a-9267-d4216673f3ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m"
Sep 17 20:48:57.368085 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.367478    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbwdg\" (UniqueName: \"kubernetes.io/projected/7d501871-a063-4d0a-9267-d4216673f3ef-kube-api-access-zbwdg\") pod \"ovnkube-node-tmg6m\" (UID: \"7d501871-a063-4d0a-9267-d4216673f3ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m"
Sep 17 20:48:57.368085 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.367535    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd-host-run-k8s-cni-cncf-io\") pod \"multus-r58z7\" (UID: \"a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd\") " pod="openshift-multus/multus-r58z7"
Sep 17 20:48:57.369105 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.367565    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/efccf4d3-5793-4300-ac5f-ea35e453da2f-etc-sysctl-d\") pod \"tuned-rk9ss\" (UID: \"efccf4d3-5793-4300-ac5f-ea35e453da2f\") " pod="openshift-cluster-node-tuning-operator/tuned-rk9ss"
Sep 17 20:48:57.369105 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.367598    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd-host-run-k8s-cni-cncf-io\") pod \"multus-r58z7\" (UID: \"a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd\") " pod="openshift-multus/multus-r58z7"
Sep 17 20:48:57.369105 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.367661    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ddbfff5-cd86-477e-9152-7b0969bb0054-system-cni-dir\") pod \"multus-additional-cni-plugins-zcsbp\" (UID: \"5ddbfff5-cd86-477e-9152-7b0969bb0054\") " pod="openshift-multus/multus-additional-cni-plugins-zcsbp"
Sep 17 20:48:57.369105 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.367688    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7d501871-a063-4d0a-9267-d4216673f3ef-host-kubelet\") pod \"ovnkube-node-tmg6m\" (UID: \"7d501871-a063-4d0a-9267-d4216673f3ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m"
Sep 17 20:48:57.369105 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.367708    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d501871-a063-4d0a-9267-d4216673f3ef-host-run-ovn-kubernetes\") pod \"ovnkube-node-tmg6m\" (UID: \"7d501871-a063-4d0a-9267-d4216673f3ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m"
Sep 17 20:48:57.369105 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.367711    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/efccf4d3-5793-4300-ac5f-ea35e453da2f-etc-sysctl-d\") pod \"tuned-rk9ss\" (UID: \"efccf4d3-5793-4300-ac5f-ea35e453da2f\") " pod="openshift-cluster-node-tuning-operator/tuned-rk9ss"
Sep 17 20:48:57.369105 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.367728    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f3d70fb8-dd97-4d0e-99b1-694aa64834ca-etc-selinux\") pod \"aws-ebs-csi-driver-node-cj2s8\" (UID: \"f3d70fb8-dd97-4d0e-99b1-694aa64834ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cj2s8"
Sep 17 20:48:57.369105 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.367745    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzpbd\" (UniqueName: \"kubernetes.io/projected/f3d70fb8-dd97-4d0e-99b1-694aa64834ca-kube-api-access-qzpbd\") pod \"aws-ebs-csi-driver-node-cj2s8\" (UID: \"f3d70fb8-dd97-4d0e-99b1-694aa64834ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cj2s8"
Sep 17 20:48:57.369105 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.367785    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/77f9970b-70e4-446e-954f-a37db50b183d-host-slash\") pod \"iptables-alerter-c4h4p\" (UID: \"77f9970b-70e4-446e-954f-a37db50b183d\") " pod="openshift-network-operator/iptables-alerter-c4h4p"
Sep 17 20:48:57.369105 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.367828    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5ddbfff5-cd86-477e-9152-7b0969bb0054-cni-binary-copy\") pod \"multus-additional-cni-plugins-zcsbp\" (UID: \"5ddbfff5-cd86-477e-9152-7b0969bb0054\") " pod="openshift-multus/multus-additional-cni-plugins-zcsbp"
Sep 17 20:48:57.369105 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.367837    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/77f9970b-70e4-446e-954f-a37db50b183d-host-slash\") pod \"iptables-alerter-c4h4p\" (UID: \"77f9970b-70e4-446e-954f-a37db50b183d\") " pod="openshift-network-operator/iptables-alerter-c4h4p"
Sep 17 20:48:57.369105 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.367872    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd-multus-daemon-config\") pod \"multus-r58z7\" (UID: \"a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd\") " pod="openshift-multus/multus-r58z7"
Sep 17 20:48:57.369105 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.367899    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/efccf4d3-5793-4300-ac5f-ea35e453da2f-sys\") pod \"tuned-rk9ss\" (UID: \"efccf4d3-5793-4300-ac5f-ea35e453da2f\") " pod="openshift-cluster-node-tuning-operator/tuned-rk9ss"
Sep 17 20:48:57.369105 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.367926    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/77f9970b-70e4-446e-954f-a37db50b183d-iptables-alerter-script\") pod \"iptables-alerter-c4h4p\" (UID: \"77f9970b-70e4-446e-954f-a37db50b183d\") " pod="openshift-network-operator/iptables-alerter-c4h4p"
Sep 17 20:48:57.369105 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.367952    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d501871-a063-4d0a-9267-d4216673f3ef-etc-openvswitch\") pod \"ovnkube-node-tmg6m\" (UID: \"7d501871-a063-4d0a-9267-d4216673f3ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m"
Sep 17 20:48:57.369105 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.367977    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/efccf4d3-5793-4300-ac5f-ea35e453da2f-sys\") pod \"tuned-rk9ss\" (UID: \"efccf4d3-5793-4300-ac5f-ea35e453da2f\") " pod="openshift-cluster-node-tuning-operator/tuned-rk9ss"
Sep 17 20:48:57.369105 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.368013    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd-host-var-lib-kubelet\") pod \"multus-r58z7\" (UID: \"a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd\") " pod="openshift-multus/multus-r58z7"
Sep 17 20:48:57.370003 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.368040    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd-multus-conf-dir\") pod \"multus-r58z7\" (UID: \"a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd\") " pod="openshift-multus/multus-r58z7"
Sep 17 20:48:57.370003 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.368073    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/efccf4d3-5793-4300-ac5f-ea35e453da2f-etc-modprobe-d\") pod \"tuned-rk9ss\" (UID: \"efccf4d3-5793-4300-ac5f-ea35e453da2f\") " pod="openshift-cluster-node-tuning-operator/tuned-rk9ss"
Sep 17 20:48:57.370003 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.368102    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7d501871-a063-4d0a-9267-d4216673f3ef-host-slash\") pod \"ovnkube-node-tmg6m\" (UID: \"7d501871-a063-4d0a-9267-d4216673f3ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m"
Sep 17 20:48:57.370003 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.368127    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d501871-a063-4d0a-9267-d4216673f3ef-var-lib-openvswitch\") pod \"ovnkube-node-tmg6m\" (UID: \"7d501871-a063-4d0a-9267-d4216673f3ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m"
Sep 17 20:48:57.370003 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.368154    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d501871-a063-4d0a-9267-d4216673f3ef-run-openvswitch\") pod \"ovnkube-node-tmg6m\" (UID: \"7d501871-a063-4d0a-9267-d4216673f3ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m"
Sep 17 20:48:57.370003 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.368182    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd-host-var-lib-cni-multus\") pod \"multus-r58z7\" (UID: \"a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd\") " pod="openshift-multus/multus-r58z7"
Sep 17 20:48:57.370003 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.368200    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/efccf4d3-5793-4300-ac5f-ea35e453da2f-etc-modprobe-d\") pod \"tuned-rk9ss\" (UID: \"efccf4d3-5793-4300-ac5f-ea35e453da2f\") " pod="openshift-cluster-node-tuning-operator/tuned-rk9ss"
Sep 17 20:48:57.370003 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.368209    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/efccf4d3-5793-4300-ac5f-ea35e453da2f-etc-systemd\") pod \"tuned-rk9ss\" (UID: \"efccf4d3-5793-4300-ac5f-ea35e453da2f\") " pod="openshift-cluster-node-tuning-operator/tuned-rk9ss"
Sep 17 20:48:57.370003 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.368235    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5ddbfff5-cd86-477e-9152-7b0969bb0054-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zcsbp\" (UID: \"5ddbfff5-cd86-477e-9152-7b0969bb0054\") " pod="openshift-multus/multus-additional-cni-plugins-zcsbp"
Sep 17 20:48:57.370003 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.368245    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd-host-var-lib-kubelet\") pod \"multus-r58z7\" (UID: \"a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd\") " pod="openshift-multus/multus-r58z7"
Sep 17 20:48:57.370003 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.368264    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f3d70fb8-dd97-4d0e-99b1-694aa64834ca-socket-dir\") pod \"aws-ebs-csi-driver-node-cj2s8\" (UID: \"f3d70fb8-dd97-4d0e-99b1-694aa64834ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cj2s8"
Sep 17 20:48:57.370003 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.368299    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72wqn\" (UniqueName: \"kubernetes.io/projected/a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd-kube-api-access-72wqn\") pod \"multus-r58z7\" (UID: \"a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd\") " pod="openshift-multus/multus-r58z7"
Sep 17 20:48:57.370003 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.368308    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd-multus-conf-dir\") pod \"multus-r58z7\" (UID: \"a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd\") " pod="openshift-multus/multus-r58z7"
Sep 17 20:48:57.370003 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.368329    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqcxv\" (UniqueName: \"kubernetes.io/projected/53dee621-a806-480a-80c6-ff174deb8ed4-kube-api-access-fqcxv\") pod \"network-check-target-l28wp\" (UID: \"53dee621-a806-480a-80c6-ff174deb8ed4\") " pod="openshift-network-diagnostics/network-check-target-l28wp"
Sep 17 20:48:57.370003 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.368356    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd-host-var-lib-cni-multus\") pod \"multus-r58z7\" (UID: \"a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd\") " pod="openshift-multus/multus-r58z7"
Sep 17 20:48:57.370003 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.368358    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfqrm\" (UniqueName: \"kubernetes.io/projected/5af35da6-3bf6-49ca-9a78-0218c7166c83-kube-api-access-nfqrm\") pod \"node-resolver-48p9f\" (UID: \"5af35da6-3bf6-49ca-9a78-0218c7166c83\") " pod="openshift-dns/node-resolver-48p9f"
Sep 17 20:48:57.370003 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.368399    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5ddbfff5-cd86-477e-9152-7b0969bb0054-cnibin\") pod \"multus-additional-cni-plugins-zcsbp\" (UID: \"5ddbfff5-cd86-477e-9152-7b0969bb0054\") " pod="openshift-multus/multus-additional-cni-plugins-zcsbp"
Sep 17 20:48:57.370975 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.368427    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7d501871-a063-4d0a-9267-d4216673f3ef-ovnkube-script-lib\") pod \"ovnkube-node-tmg6m\" (UID: \"7d501871-a063-4d0a-9267-d4216673f3ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m"
Sep 17 20:48:57.370975 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.368453    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0369efa0-b31a-4fa5-b426-95ba067ea731-serviceca\") pod \"node-ca-2lrgx\" (UID: \"0369efa0-b31a-4fa5-b426-95ba067ea731\") " pod="openshift-image-registry/node-ca-2lrgx"
Sep 17 20:48:57.370975 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.368484    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/77f9970b-70e4-446e-954f-a37db50b183d-iptables-alerter-script\") pod \"iptables-alerter-c4h4p\" (UID: \"77f9970b-70e4-446e-954f-a37db50b183d\") " pod="openshift-network-operator/iptables-alerter-c4h4p"
Sep 17 20:48:57.370975 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.368484    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd-cni-binary-copy\") pod \"multus-r58z7\" (UID: \"a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd\") " pod="openshift-multus/multus-r58z7"
Sep 17 20:48:57.370975 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.368501    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd-multus-daemon-config\") pod \"multus-r58z7\" (UID: \"a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd\") " pod="openshift-multus/multus-r58z7"
Sep 17 20:48:57.370975 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.368566    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/efccf4d3-5793-4300-ac5f-ea35e453da2f-etc-systemd\") pod \"tuned-rk9ss\" (UID: \"efccf4d3-5793-4300-ac5f-ea35e453da2f\") " pod="openshift-cluster-node-tuning-operator/tuned-rk9ss"
Sep 17 20:48:57.370975 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.368614    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd-multus-socket-dir-parent\") pod \"multus-r58z7\" (UID: \"a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd\") " pod="openshift-multus/multus-r58z7"
Sep 17 20:48:57.370975 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.368678    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/efccf4d3-5793-4300-ac5f-ea35e453da2f-etc-sysconfig\") pod \"tuned-rk9ss\" (UID: \"efccf4d3-5793-4300-ac5f-ea35e453da2f\") " pod="openshift-cluster-node-tuning-operator/tuned-rk9ss"
Sep 17 20:48:57.370975 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.368688    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd-multus-socket-dir-parent\") pod \"multus-r58z7\" (UID: \"a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd\") " pod="openshift-multus/multus-r58z7"
Sep 17 20:48:57.370975 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.368794    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/efccf4d3-5793-4300-ac5f-ea35e453da2f-run\") pod \"tuned-rk9ss\" (UID: \"efccf4d3-5793-4300-ac5f-ea35e453da2f\") " pod="openshift-cluster-node-tuning-operator/tuned-rk9ss"
Sep 17 20:48:57.370975 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.368826    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/efccf4d3-5793-4300-ac5f-ea35e453da2f-etc-sysconfig\") pod \"tuned-rk9ss\" (UID: \"efccf4d3-5793-4300-ac5f-ea35e453da2f\") " pod="openshift-cluster-node-tuning-operator/tuned-rk9ss"
Sep 17 20:48:57.370975 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.368840    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5af35da6-3bf6-49ca-9a78-0218c7166c83-hosts-file\") pod \"node-resolver-48p9f\" (UID: \"5af35da6-3bf6-49ca-9a78-0218c7166c83\") " pod="openshift-dns/node-resolver-48p9f"
Sep 17 20:48:57.370975 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.368894    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/efccf4d3-5793-4300-ac5f-ea35e453da2f-run\") pod \"tuned-rk9ss\" (UID: \"efccf4d3-5793-4300-ac5f-ea35e453da2f\") " pod="openshift-cluster-node-tuning-operator/tuned-rk9ss"
Sep 17 20:48:57.370975 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.368912    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5af35da6-3bf6-49ca-9a78-0218c7166c83-tmp-dir\") pod \"node-resolver-48p9f\" (UID: \"5af35da6-3bf6-49ca-9a78-0218c7166c83\") " pod="openshift-dns/node-resolver-48p9f"
Sep 17 20:48:57.370975 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.368947    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5af35da6-3bf6-49ca-9a78-0218c7166c83-hosts-file\") pod \"node-resolver-48p9f\" (UID: \"5af35da6-3bf6-49ca-9a78-0218c7166c83\") " pod="openshift-dns/node-resolver-48p9f"
Sep 17 20:48:57.370975 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.369037    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd-cni-binary-copy\") pod \"multus-r58z7\" (UID: \"a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd\") " pod="openshift-multus/multus-r58z7"
Sep 17 20:48:57.370975 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.369271    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5af35da6-3bf6-49ca-9a78-0218c7166c83-tmp-dir\") pod \"node-resolver-48p9f\" (UID: \"5af35da6-3bf6-49ca-9a78-0218c7166c83\") " pod="openshift-dns/node-resolver-48p9f"
Sep 17 20:48:57.374398 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.374368    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/efccf4d3-5793-4300-ac5f-ea35e453da2f-etc-tuned\") pod \"tuned-rk9ss\" (UID: \"efccf4d3-5793-4300-ac5f-ea35e453da2f\") " pod="openshift-cluster-node-tuning-operator/tuned-rk9ss"
Sep 17 20:48:57.375116 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.375092    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/efccf4d3-5793-4300-ac5f-ea35e453da2f-tmp\") pod \"tuned-rk9ss\" (UID: \"efccf4d3-5793-4300-ac5f-ea35e453da2f\") " pod="openshift-cluster-node-tuning-operator/tuned-rk9ss"
Sep 17 20:48:57.379715 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:48:57.379445    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 20:48:57.379715 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:48:57.379469    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 20:48:57.379715 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:48:57.379487    2575 projected.go:196] Error preparing data for projected volume kube-api-access-fqcxv for pod openshift-network-diagnostics/network-check-target-l28wp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:48:57.379715 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:48:57.379555    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/53dee621-a806-480a-80c6-ff174deb8ed4-kube-api-access-fqcxv podName:53dee621-a806-480a-80c6-ff174deb8ed4 nodeName:}" failed. No retries permitted until 2026-09-17 20:48:57.879533943 +0000 UTC m=+3.031978307 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-fqcxv" (UniqueName: "kubernetes.io/projected/53dee621-a806-480a-80c6-ff174deb8ed4-kube-api-access-fqcxv") pod "network-check-target-l28wp" (UID: "53dee621-a806-480a-80c6-ff174deb8ed4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:48:57.380004 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.379793    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrt5f\" (UniqueName: \"kubernetes.io/projected/efccf4d3-5793-4300-ac5f-ea35e453da2f-kube-api-access-hrt5f\") pod \"tuned-rk9ss\" (UID: \"efccf4d3-5793-4300-ac5f-ea35e453da2f\") " pod="openshift-cluster-node-tuning-operator/tuned-rk9ss"
Sep 17 20:48:57.380530 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.380486    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4grcw\" (UniqueName: \"kubernetes.io/projected/77f9970b-70e4-446e-954f-a37db50b183d-kube-api-access-4grcw\") pod \"iptables-alerter-c4h4p\" (UID: \"77f9970b-70e4-446e-954f-a37db50b183d\") " pod="openshift-network-operator/iptables-alerter-c4h4p"
Sep 17 20:48:57.382735 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.382640    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9vzl\" (UniqueName: \"kubernetes.io/projected/d42a790e-18b7-4478-b0c7-3c7cc4372fff-kube-api-access-l9vzl\") pod \"network-metrics-daemon-8b6dt\" (UID: \"d42a790e-18b7-4478-b0c7-3c7cc4372fff\") " pod="openshift-multus/network-metrics-daemon-8b6dt"
Sep 17 20:48:57.383130 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.383073    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72wqn\" (UniqueName: \"kubernetes.io/projected/a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd-kube-api-access-72wqn\") pod \"multus-r58z7\" (UID: \"a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd\") " pod="openshift-multus/multus-r58z7"
Sep 17 20:48:57.385707 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.385661    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfqrm\" (UniqueName: \"kubernetes.io/projected/5af35da6-3bf6-49ca-9a78-0218c7166c83-kube-api-access-nfqrm\") pod \"node-resolver-48p9f\" (UID: \"5af35da6-3bf6-49ca-9a78-0218c7166c83\") " pod="openshift-dns/node-resolver-48p9f"
Sep 17 20:48:57.469985 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.469850    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5ddbfff5-cd86-477e-9152-7b0969bb0054-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zcsbp\" (UID: \"5ddbfff5-cd86-477e-9152-7b0969bb0054\") " pod="openshift-multus/multus-additional-cni-plugins-zcsbp"
Sep 17 20:48:57.469985 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.469920    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f3d70fb8-dd97-4d0e-99b1-694aa64834ca-socket-dir\") pod \"aws-ebs-csi-driver-node-cj2s8\" (UID: \"f3d70fb8-dd97-4d0e-99b1-694aa64834ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cj2s8"
Sep 17 20:48:57.470242 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.469992    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5ddbfff5-cd86-477e-9152-7b0969bb0054-cnibin\") pod \"multus-additional-cni-plugins-zcsbp\" (UID: \"5ddbfff5-cd86-477e-9152-7b0969bb0054\") " pod="openshift-multus/multus-additional-cni-plugins-zcsbp"
Sep 17 20:48:57.470242 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.470036    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7d501871-a063-4d0a-9267-d4216673f3ef-ovnkube-script-lib\") pod \"ovnkube-node-tmg6m\" (UID: \"7d501871-a063-4d0a-9267-d4216673f3ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m"
Sep 17 20:48:57.470242 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.470064    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0369efa0-b31a-4fa5-b426-95ba067ea731-serviceca\") pod \"node-ca-2lrgx\" (UID: \"0369efa0-b31a-4fa5-b426-95ba067ea731\") " pod="openshift-image-registry/node-ca-2lrgx"
Sep 17 20:48:57.470242 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.470096    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7d501871-a063-4d0a-9267-d4216673f3ef-host-run-netns\") pod \"ovnkube-node-tmg6m\" (UID: \"7d501871-a063-4d0a-9267-d4216673f3ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m"
Sep 17 20:48:57.470242 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.470101    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f3d70fb8-dd97-4d0e-99b1-694aa64834ca-socket-dir\") pod \"aws-ebs-csi-driver-node-cj2s8\" (UID: \"f3d70fb8-dd97-4d0e-99b1-694aa64834ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cj2s8"
Sep 17 20:48:57.470242 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.470121    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7d501871-a063-4d0a-9267-d4216673f3ef-node-log\") pod \"ovnkube-node-tmg6m\" (UID: \"7d501871-a063-4d0a-9267-d4216673f3ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m"
Sep 17 20:48:57.470242 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.470150    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/334a13ac-ccb2-43eb-bd43-566981c058ff-kubelet-config\") pod \"global-pull-secret-syncer-dpwhq\" (UID: \"334a13ac-ccb2-43eb-bd43-566981c058ff\") " pod="kube-system/global-pull-secret-syncer-dpwhq"
Sep 17 20:48:57.470242 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.470177    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5ddbfff5-cd86-477e-9152-7b0969bb0054-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zcsbp\" (UID: \"5ddbfff5-cd86-477e-9152-7b0969bb0054\") " pod="openshift-multus/multus-additional-cni-plugins-zcsbp"
Sep 17 20:48:57.470242 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.470173    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5ddbfff5-cd86-477e-9152-7b0969bb0054-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zcsbp\" (UID: \"5ddbfff5-cd86-477e-9152-7b0969bb0054\") " pod="openshift-multus/multus-additional-cni-plugins-zcsbp"
Sep 17 20:48:57.470242 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.470225    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7d501871-a063-4d0a-9267-d4216673f3ef-host-run-netns\") pod \"ovnkube-node-tmg6m\" (UID: \"7d501871-a063-4d0a-9267-d4216673f3ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m"
Sep 17 20:48:57.470242 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.470245    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7d501871-a063-4d0a-9267-d4216673f3ef-node-log\") pod \"ovnkube-node-tmg6m\" (UID: \"7d501871-a063-4d0a-9267-d4216673f3ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m"
Sep 17 20:48:57.470242 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.470264    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d501871-a063-4d0a-9267-d4216673f3ef-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tmg6m\" (UID: \"7d501871-a063-4d0a-9267-d4216673f3ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m"
Sep 17 20:48:57.470938 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.470289    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/334a13ac-ccb2-43eb-bd43-566981c058ff-kubelet-config\") pod \"global-pull-secret-syncer-dpwhq\" (UID: \"334a13ac-ccb2-43eb-bd43-566981c058ff\") " pod="kube-system/global-pull-secret-syncer-dpwhq"
Sep 17 20:48:57.470938 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.470290    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5ddbfff5-cd86-477e-9152-7b0969bb0054-cnibin\") pod \"multus-additional-cni-plugins-zcsbp\" (UID: \"5ddbfff5-cd86-477e-9152-7b0969bb0054\") " pod="openshift-multus/multus-additional-cni-plugins-zcsbp"
Sep 17 20:48:57.470938 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.470360    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d501871-a063-4d0a-9267-d4216673f3ef-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tmg6m\" (UID: \"7d501871-a063-4d0a-9267-d4216673f3ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m"
Sep 17 20:48:57.470938 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.470300    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7d501871-a063-4d0a-9267-d4216673f3ef-ovnkube-config\") pod \"ovnkube-node-tmg6m\" (UID: \"7d501871-a063-4d0a-9267-d4216673f3ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m"
Sep 17 20:48:57.470938 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.470405    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f3d70fb8-dd97-4d0e-99b1-694aa64834ca-registration-dir\") pod \"aws-ebs-csi-driver-node-cj2s8\" (UID: \"f3d70fb8-dd97-4d0e-99b1-694aa64834ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cj2s8"
Sep 17 20:48:57.470938 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.470438    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/334a13ac-ccb2-43eb-bd43-566981c058ff-dbus\") pod \"global-pull-secret-syncer-dpwhq\" (UID: \"334a13ac-ccb2-43eb-bd43-566981c058ff\") " pod="kube-system/global-pull-secret-syncer-dpwhq"
Sep 17 20:48:57.470938 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.470467    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7d501871-a063-4d0a-9267-d4216673f3ef-run-systemd\") pod \"ovnkube-node-tmg6m\" (UID: \"7d501871-a063-4d0a-9267-d4216673f3ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m"
Sep 17 20:48:57.470938 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.470496    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7d501871-a063-4d0a-9267-d4216673f3ef-run-ovn\") pod \"ovnkube-node-tmg6m\" (UID: \"7d501871-a063-4d0a-9267-d4216673f3ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m"
Sep 17 20:48:57.470938 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.470520    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7d501871-a063-4d0a-9267-d4216673f3ef-log-socket\") pod \"ovnkube-node-tmg6m\" (UID: \"7d501871-a063-4d0a-9267-d4216673f3ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m"
Sep 17 20:48:57.470938 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.470549    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7d501871-a063-4d0a-9267-d4216673f3ef-ovn-node-metrics-cert\") pod \"ovnkube-node-tmg6m\" (UID: \"7d501871-a063-4d0a-9267-d4216673f3ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m"
Sep 17 20:48:57.470938 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.470574    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f3d70fb8-dd97-4d0e-99b1-694aa64834ca-sys-fs\") pod \"aws-ebs-csi-driver-node-cj2s8\" (UID: \"f3d70fb8-dd97-4d0e-99b1-694aa64834ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cj2s8"
Sep 17 20:48:57.470938 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.470601    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0369efa0-b31a-4fa5-b426-95ba067ea731-host\") pod \"node-ca-2lrgx\" (UID: \"0369efa0-b31a-4fa5-b426-95ba067ea731\") " pod="openshift-image-registry/node-ca-2lrgx"
Sep 17 20:48:57.470938 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.470628    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ftff\" (UniqueName: \"kubernetes.io/projected/0369efa0-b31a-4fa5-b426-95ba067ea731-kube-api-access-4ftff\") pod \"node-ca-2lrgx\" (UID: \"0369efa0-b31a-4fa5-b426-95ba067ea731\") " pod="openshift-image-registry/node-ca-2lrgx"
Sep 17 20:48:57.470938 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.470658    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt2tb\" (UniqueName: \"kubernetes.io/projected/5ddbfff5-cd86-477e-9152-7b0969bb0054-kube-api-access-lt2tb\") pod \"multus-additional-cni-plugins-zcsbp\" (UID: \"5ddbfff5-cd86-477e-9152-7b0969bb0054\") " pod="openshift-multus/multus-additional-cni-plugins-zcsbp"
Sep 17 20:48:57.470938 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.470684    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7d501871-a063-4d0a-9267-d4216673f3ef-systemd-units\") pod \"ovnkube-node-tmg6m\" (UID: \"7d501871-a063-4d0a-9267-d4216673f3ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m"
Sep 17 20:48:57.470938 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.470710    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7d501871-a063-4d0a-9267-d4216673f3ef-env-overrides\") pod \"ovnkube-node-tmg6m\" (UID: \"7d501871-a063-4d0a-9267-d4216673f3ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m"
Sep 17 20:48:57.470938 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.470734    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f3d70fb8-dd97-4d0e-99b1-694aa64834ca-device-dir\") pod \"aws-ebs-csi-driver-node-cj2s8\" (UID: \"f3d70fb8-dd97-4d0e-99b1-694aa64834ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cj2s8"
Sep 17 20:48:57.471885 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.470761    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/334a13ac-ccb2-43eb-bd43-566981c058ff-original-pull-secret\") pod \"global-pull-secret-syncer-dpwhq\" (UID: \"334a13ac-ccb2-43eb-bd43-566981c058ff\") " pod="kube-system/global-pull-secret-syncer-dpwhq"
Sep 17 20:48:57.471885 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.470787    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7d501871-a063-4d0a-9267-d4216673f3ef-host-cni-bin\") pod \"ovnkube-node-tmg6m\" (UID: \"7d501871-a063-4d0a-9267-d4216673f3ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m"
Sep 17 20:48:57.471885 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.470796    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7d501871-a063-4d0a-9267-d4216673f3ef-ovnkube-config\") pod \"ovnkube-node-tmg6m\" (UID: \"7d501871-a063-4d0a-9267-d4216673f3ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m"
Sep 17 20:48:57.471885 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.470806    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0369efa0-b31a-4fa5-b426-95ba067ea731-serviceca\") pod \"node-ca-2lrgx\" (UID: \"0369efa0-b31a-4fa5-b426-95ba067ea731\") " pod="openshift-image-registry/node-ca-2lrgx"
Sep 17 20:48:57.471885 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.470814    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1a25b3d5-790d-41a8-bea7-4689843c7d71-agent-certs\") pod \"konnectivity-agent-mk66j\" (UID: \"1a25b3d5-790d-41a8-bea7-4689843c7d71\") " pod="kube-system/konnectivity-agent-mk66j"
Sep 17 20:48:57.471885 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.470840    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f3d70fb8-dd97-4d0e-99b1-694aa64834ca-sys-fs\") pod \"aws-ebs-csi-driver-node-cj2s8\" (UID: \"f3d70fb8-dd97-4d0e-99b1-694aa64834ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cj2s8"
Sep 17 20:48:57.471885 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.470855    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7d501871-a063-4d0a-9267-d4216673f3ef-run-systemd\") pod \"ovnkube-node-tmg6m\" (UID: \"7d501871-a063-4d0a-9267-d4216673f3ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m"
Sep 17 20:48:57.471885 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.470841    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1a25b3d5-790d-41a8-bea7-4689843c7d71-konnectivity-ca\") pod \"konnectivity-agent-mk66j\" (UID: \"1a25b3d5-790d-41a8-bea7-4689843c7d71\") " pod="kube-system/konnectivity-agent-mk66j"
Sep 17 20:48:57.471885 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.470886    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0369efa0-b31a-4fa5-b426-95ba067ea731-host\") pod \"node-ca-2lrgx\" (UID: \"0369efa0-b31a-4fa5-b426-95ba067ea731\") " pod="openshift-image-registry/node-ca-2lrgx"
Sep 17 20:48:57.471885 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.470863    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7d501871-a063-4d0a-9267-d4216673f3ef-log-socket\") pod \"ovnkube-node-tmg6m\" (UID: \"7d501871-a063-4d0a-9267-d4216673f3ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m"
Sep 17 20:48:57.471885 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.470785    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7d501871-a063-4d0a-9267-d4216673f3ef-ovnkube-script-lib\") pod \"ovnkube-node-tmg6m\" (UID: \"7d501871-a063-4d0a-9267-d4216673f3ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m"
Sep 17 20:48:57.471885 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.470924    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f3d70fb8-dd97-4d0e-99b1-694aa64834ca-device-dir\") pod \"aws-ebs-csi-driver-node-cj2s8\" (UID: \"f3d70fb8-dd97-4d0e-99b1-694aa64834ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cj2s8"
Sep 17 20:48:57.471885 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.470896    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f3d70fb8-dd97-4d0e-99b1-694aa64834ca-registration-dir\") pod \"aws-ebs-csi-driver-node-cj2s8\" (UID: \"f3d70fb8-dd97-4d0e-99b1-694aa64834ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cj2s8"
Sep 17 20:48:57.471885 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.470847    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5ddbfff5-cd86-477e-9152-7b0969bb0054-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zcsbp\" (UID: \"5ddbfff5-cd86-477e-9152-7b0969bb0054\") " pod="openshift-multus/multus-additional-cni-plugins-zcsbp"
Sep 17 20:48:57.471885 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.470896    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7d501871-a063-4d0a-9267-d4216673f3ef-run-ovn\") pod \"ovnkube-node-tmg6m\" (UID: \"7d501871-a063-4d0a-9267-d4216673f3ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m"
Sep 17 20:48:57.471885 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.470905    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5ddbfff5-cd86-477e-9152-7b0969bb0054-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zcsbp\" (UID: \"5ddbfff5-cd86-477e-9152-7b0969bb0054\") " pod="openshift-multus/multus-additional-cni-plugins-zcsbp"
Sep 17 20:48:57.471885 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.470992    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/334a13ac-ccb2-43eb-bd43-566981c058ff-dbus\") pod \"global-pull-secret-syncer-dpwhq\" (UID: \"334a13ac-ccb2-43eb-bd43-566981c058ff\") " pod="kube-system/global-pull-secret-syncer-dpwhq"
Sep 17 20:48:57.472744 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.470989    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f3d70fb8-dd97-4d0e-99b1-694aa64834ca-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cj2s8\" (UID: \"f3d70fb8-dd97-4d0e-99b1-694aa64834ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cj2s8"
Sep 17 20:48:57.472744 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:48:57.471077    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 20:48:57.472744 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.471114    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7d501871-a063-4d0a-9267-d4216673f3ef-host-cni-bin\") pod \"ovnkube-node-tmg6m\" (UID: \"7d501871-a063-4d0a-9267-d4216673f3ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m"
Sep 17 20:48:57.472744 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.471137    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7d501871-a063-4d0a-9267-d4216673f3ef-host-cni-netd\") pod \"ovnkube-node-tmg6m\" (UID: \"7d501871-a063-4d0a-9267-d4216673f3ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m"
Sep 17 20:48:57.472744 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.471173    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbwdg\" (UniqueName: \"kubernetes.io/projected/7d501871-a063-4d0a-9267-d4216673f3ef-kube-api-access-zbwdg\") pod \"ovnkube-node-tmg6m\" (UID: \"7d501871-a063-4d0a-9267-d4216673f3ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m"
Sep 17 20:48:57.472744 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.471187    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7d501871-a063-4d0a-9267-d4216673f3ef-host-cni-netd\") pod \"ovnkube-node-tmg6m\" (UID: \"7d501871-a063-4d0a-9267-d4216673f3ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m"
Sep 17 20:48:57.472744 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.471205    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ddbfff5-cd86-477e-9152-7b0969bb0054-system-cni-dir\") pod \"multus-additional-cni-plugins-zcsbp\" (UID: \"5ddbfff5-cd86-477e-9152-7b0969bb0054\") " pod="openshift-multus/multus-additional-cni-plugins-zcsbp"
Sep 17 20:48:57.472744 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:48:57.471239    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/334a13ac-ccb2-43eb-bd43-566981c058ff-original-pull-secret podName:334a13ac-ccb2-43eb-bd43-566981c058ff nodeName:}" failed. No retries permitted until 2026-09-17 20:48:57.971217979 +0000 UTC m=+3.123662345 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/334a13ac-ccb2-43eb-bd43-566981c058ff-original-pull-secret") pod "global-pull-secret-syncer-dpwhq" (UID: "334a13ac-ccb2-43eb-bd43-566981c058ff") : object "kube-system"/"original-pull-secret" not registered
Sep 17 20:48:57.472744 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.471256    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ddbfff5-cd86-477e-9152-7b0969bb0054-system-cni-dir\") pod \"multus-additional-cni-plugins-zcsbp\" (UID: \"5ddbfff5-cd86-477e-9152-7b0969bb0054\") " pod="openshift-multus/multus-additional-cni-plugins-zcsbp"
Sep 17 20:48:57.472744 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.471259    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7d501871-a063-4d0a-9267-d4216673f3ef-host-kubelet\") pod \"ovnkube-node-tmg6m\" (UID: \"7d501871-a063-4d0a-9267-d4216673f3ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m"
Sep 17 20:48:57.472744 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.471299    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7d501871-a063-4d0a-9267-d4216673f3ef-host-kubelet\") pod \"ovnkube-node-tmg6m\" (UID: \"7d501871-a063-4d0a-9267-d4216673f3ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m"
Sep 17 20:48:57.472744 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.471297    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d501871-a063-4d0a-9267-d4216673f3ef-host-run-ovn-kubernetes\") pod \"ovnkube-node-tmg6m\" (UID: \"7d501871-a063-4d0a-9267-d4216673f3ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m"
Sep 17 20:48:57.472744 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.471335    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d501871-a063-4d0a-9267-d4216673f3ef-host-run-ovn-kubernetes\") pod \"ovnkube-node-tmg6m\" (UID: \"7d501871-a063-4d0a-9267-d4216673f3ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m"
Sep 17 20:48:57.472744 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.471340    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f3d70fb8-dd97-4d0e-99b1-694aa64834ca-etc-selinux\") pod \"aws-ebs-csi-driver-node-cj2s8\" (UID: \"f3d70fb8-dd97-4d0e-99b1-694aa64834ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cj2s8"
Sep 17 20:48:57.472744 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.471376    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzpbd\" (UniqueName: \"kubernetes.io/projected/f3d70fb8-dd97-4d0e-99b1-694aa64834ca-kube-api-access-qzpbd\") pod \"aws-ebs-csi-driver-node-cj2s8\" (UID: \"f3d70fb8-dd97-4d0e-99b1-694aa64834ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cj2s8"
Sep 17 20:48:57.472744 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.471407    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5ddbfff5-cd86-477e-9152-7b0969bb0054-cni-binary-copy\") pod \"multus-additional-cni-plugins-zcsbp\" (UID: \"5ddbfff5-cd86-477e-9152-7b0969bb0054\") " pod="openshift-multus/multus-additional-cni-plugins-zcsbp"
Sep 17 20:48:57.472744 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.471438    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d501871-a063-4d0a-9267-d4216673f3ef-etc-openvswitch\") pod \"ovnkube-node-tmg6m\" (UID: \"7d501871-a063-4d0a-9267-d4216673f3ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m"
Sep 17 20:48:57.473745 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.471465    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7d501871-a063-4d0a-9267-d4216673f3ef-host-slash\") pod \"ovnkube-node-tmg6m\" (UID: \"7d501871-a063-4d0a-9267-d4216673f3ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m"
Sep 17 20:48:57.473745 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.471497    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d501871-a063-4d0a-9267-d4216673f3ef-var-lib-openvswitch\") pod \"ovnkube-node-tmg6m\" (UID: \"7d501871-a063-4d0a-9267-d4216673f3ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m"
Sep 17 20:48:57.473745 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.471522    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d501871-a063-4d0a-9267-d4216673f3ef-run-openvswitch\") pod \"ovnkube-node-tmg6m\" (UID: \"7d501871-a063-4d0a-9267-d4216673f3ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m"
Sep 17 20:48:57.473745 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.471521    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7d501871-a063-4d0a-9267-d4216673f3ef-systemd-units\") pod \"ovnkube-node-tmg6m\" (UID: \"7d501871-a063-4d0a-9267-d4216673f3ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m"
Sep 17 20:48:57.473745 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.471592    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d501871-a063-4d0a-9267-d4216673f3ef-run-openvswitch\") pod \"ovnkube-node-tmg6m\" (UID: \"7d501871-a063-4d0a-9267-d4216673f3ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m"
Sep 17 20:48:57.473745 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.471633    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f3d70fb8-dd97-4d0e-99b1-694aa64834ca-etc-selinux\") pod \"aws-ebs-csi-driver-node-cj2s8\" (UID: \"f3d70fb8-dd97-4d0e-99b1-694aa64834ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cj2s8"
Sep 17 20:48:57.473745 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.471638    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1a25b3d5-790d-41a8-bea7-4689843c7d71-konnectivity-ca\") pod \"konnectivity-agent-mk66j\" (UID: \"1a25b3d5-790d-41a8-bea7-4689843c7d71\") " pod="kube-system/konnectivity-agent-mk66j"
Sep 17 20:48:57.473745 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.471665    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7d501871-a063-4d0a-9267-d4216673f3ef-host-slash\") pod \"ovnkube-node-tmg6m\" (UID: \"7d501871-a063-4d0a-9267-d4216673f3ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m"
Sep 17 20:48:57.473745 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.471704    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d501871-a063-4d0a-9267-d4216673f3ef-var-lib-openvswitch\") pod \"ovnkube-node-tmg6m\" (UID: \"7d501871-a063-4d0a-9267-d4216673f3ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m"
Sep 17 20:48:57.473745 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.471634    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d501871-a063-4d0a-9267-d4216673f3ef-etc-openvswitch\") pod \"ovnkube-node-tmg6m\" (UID: \"7d501871-a063-4d0a-9267-d4216673f3ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m"
Sep 17 20:48:57.473745 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.471851    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f3d70fb8-dd97-4d0e-99b1-694aa64834ca-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cj2s8\" (UID: \"f3d70fb8-dd97-4d0e-99b1-694aa64834ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cj2s8"
Sep 17 20:48:57.473745 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.472026    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7d501871-a063-4d0a-9267-d4216673f3ef-env-overrides\") pod \"ovnkube-node-tmg6m\" (UID: \"7d501871-a063-4d0a-9267-d4216673f3ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m"
Sep 17 20:48:57.473745 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.472049    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5ddbfff5-cd86-477e-9152-7b0969bb0054-cni-binary-copy\") pod \"multus-additional-cni-plugins-zcsbp\" (UID: \"5ddbfff5-cd86-477e-9152-7b0969bb0054\") " pod="openshift-multus/multus-additional-cni-plugins-zcsbp"
Sep 17 20:48:57.473745 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.472473    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5ddbfff5-cd86-477e-9152-7b0969bb0054-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zcsbp\" (UID: \"5ddbfff5-cd86-477e-9152-7b0969bb0054\") " pod="openshift-multus/multus-additional-cni-plugins-zcsbp"
Sep 17 20:48:57.477403 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.477338    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1a25b3d5-790d-41a8-bea7-4689843c7d71-agent-certs\") pod \"konnectivity-agent-mk66j\" (UID: \"1a25b3d5-790d-41a8-bea7-4689843c7d71\") " pod="kube-system/konnectivity-agent-mk66j"
Sep 17 20:48:57.479547 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.479504    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7d501871-a063-4d0a-9267-d4216673f3ef-ovn-node-metrics-cert\") pod \"ovnkube-node-tmg6m\" (UID: \"7d501871-a063-4d0a-9267-d4216673f3ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m"
Sep 17 20:48:57.481436 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.481391    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ftff\" (UniqueName: \"kubernetes.io/projected/0369efa0-b31a-4fa5-b426-95ba067ea731-kube-api-access-4ftff\") pod \"node-ca-2lrgx\" (UID: \"0369efa0-b31a-4fa5-b426-95ba067ea731\") " pod="openshift-image-registry/node-ca-2lrgx"
Sep 17 20:48:57.482204 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.481872    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt2tb\" (UniqueName: \"kubernetes.io/projected/5ddbfff5-cd86-477e-9152-7b0969bb0054-kube-api-access-lt2tb\") pod \"multus-additional-cni-plugins-zcsbp\" (UID: \"5ddbfff5-cd86-477e-9152-7b0969bb0054\") " pod="openshift-multus/multus-additional-cni-plugins-zcsbp"
Sep 17 20:48:57.484240 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.484214    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbwdg\" (UniqueName: \"kubernetes.io/projected/7d501871-a063-4d0a-9267-d4216673f3ef-kube-api-access-zbwdg\") pod \"ovnkube-node-tmg6m\" (UID: \"7d501871-a063-4d0a-9267-d4216673f3ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m"
Sep 17 20:48:57.486479 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.486454    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzpbd\" (UniqueName: \"kubernetes.io/projected/f3d70fb8-dd97-4d0e-99b1-694aa64834ca-kube-api-access-qzpbd\") pod \"aws-ebs-csi-driver-node-cj2s8\" (UID: \"f3d70fb8-dd97-4d0e-99b1-694aa64834ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cj2s8"
Sep 17 20:48:57.552396 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.552354    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-r58z7"
Sep 17 20:48:57.560815 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.560784    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rk9ss"
Sep 17 20:48:57.573815 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.573788    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-48p9f"
Sep 17 20:48:57.585272 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.585242    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-c4h4p"
Sep 17 20:48:57.593219 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.593197    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m"
Sep 17 20:48:57.602417 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.602394    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mk66j"
Sep 17 20:48:57.619451 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.619424    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cj2s8"
Sep 17 20:48:57.627765 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.627740    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zcsbp"
Sep 17 20:48:57.636102 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.636081    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2lrgx"
Sep 17 20:48:57.875250 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.875169    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d42a790e-18b7-4478-b0c7-3c7cc4372fff-metrics-certs\") pod \"network-metrics-daemon-8b6dt\" (UID: \"d42a790e-18b7-4478-b0c7-3c7cc4372fff\") " pod="openshift-multus/network-metrics-daemon-8b6dt"
Sep 17 20:48:57.875395 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:48:57.875325    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:48:57.875445 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:48:57.875415    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d42a790e-18b7-4478-b0c7-3c7cc4372fff-metrics-certs podName:d42a790e-18b7-4478-b0c7-3c7cc4372fff nodeName:}" failed. No retries permitted until 2026-09-17 20:48:58.875401722 +0000 UTC m=+4.027846081 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d42a790e-18b7-4478-b0c7-3c7cc4372fff-metrics-certs") pod "network-metrics-daemon-8b6dt" (UID: "d42a790e-18b7-4478-b0c7-3c7cc4372fff") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:48:57.976103 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.976059    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqcxv\" (UniqueName: \"kubernetes.io/projected/53dee621-a806-480a-80c6-ff174deb8ed4-kube-api-access-fqcxv\") pod \"network-check-target-l28wp\" (UID: \"53dee621-a806-480a-80c6-ff174deb8ed4\") " pod="openshift-network-diagnostics/network-check-target-l28wp"
Sep 17 20:48:57.976300 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:57.976126    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/334a13ac-ccb2-43eb-bd43-566981c058ff-original-pull-secret\") pod \"global-pull-secret-syncer-dpwhq\" (UID: \"334a13ac-ccb2-43eb-bd43-566981c058ff\") " pod="kube-system/global-pull-secret-syncer-dpwhq"
Sep 17 20:48:57.976300 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:48:57.976264    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 20:48:57.976300 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:48:57.976269    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 20:48:57.976300 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:48:57.976298    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 20:48:57.976500 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:48:57.976313    2575 projected.go:196] Error preparing data for projected volume kube-api-access-fqcxv for pod openshift-network-diagnostics/network-check-target-l28wp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:48:57.976500 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:48:57.976337    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/334a13ac-ccb2-43eb-bd43-566981c058ff-original-pull-secret podName:334a13ac-ccb2-43eb-bd43-566981c058ff nodeName:}" failed. No retries permitted until 2026-09-17 20:48:58.976316813 +0000 UTC m=+4.128761188 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/334a13ac-ccb2-43eb-bd43-566981c058ff-original-pull-secret") pod "global-pull-secret-syncer-dpwhq" (UID: "334a13ac-ccb2-43eb-bd43-566981c058ff") : object "kube-system"/"original-pull-secret" not registered
Sep 17 20:48:57.976500 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:48:57.976372    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/53dee621-a806-480a-80c6-ff174deb8ed4-kube-api-access-fqcxv podName:53dee621-a806-480a-80c6-ff174deb8ed4 nodeName:}" failed. No retries permitted until 2026-09-17 20:48:58.976353886 +0000 UTC m=+4.128798259 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-fqcxv" (UniqueName: "kubernetes.io/projected/53dee621-a806-480a-80c6-ff174deb8ed4-kube-api-access-fqcxv") pod "network-check-target-l28wp" (UID: "53dee621-a806-480a-80c6-ff174deb8ed4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:48:58.307202 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:58.307157    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 20:43:57 +0000 UTC" deadline="2028-06-08 16:54:23.160186826 +0000 UTC"
Sep 17 20:48:58.307202 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:58.307201    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15116h5m24.852992232s"
Sep 17 20:48:58.602289 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:58.602221    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5af35da6_3bf6_49ca_9a78_0218c7166c83.slice/crio-f6afa78fedacc0b60feea59b4a81b3d843103e32cd911b286d816467f6e0f9e9 WatchSource:0}: Error finding container f6afa78fedacc0b60feea59b4a81b3d843103e32cd911b286d816467f6e0f9e9: Status 404 returned error can't find the container with id f6afa78fedacc0b60feea59b4a81b3d843103e32cd911b286d816467f6e0f9e9
Sep 17 20:48:58.606044 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:58.605920    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0369efa0_b31a_4fa5_b426_95ba067ea731.slice/crio-f4124c6a6beb36cdd049e480875f097cef2f9a9bbd9f0d72d5dbf9f8f92a9df3 WatchSource:0}: Error finding container f4124c6a6beb36cdd049e480875f097cef2f9a9bbd9f0d72d5dbf9f8f92a9df3: Status 404 returned error can't find the container with id f4124c6a6beb36cdd049e480875f097cef2f9a9bbd9f0d72d5dbf9f8f92a9df3
Sep 17 20:48:58.609008 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:58.608571    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3d70fb8_dd97_4d0e_99b1_694aa64834ca.slice/crio-0e2acabbb80eb065f0e5466ac23127fbd3fc6d1f2e4157a2d42a8d54c0bde943 WatchSource:0}: Error finding container 0e2acabbb80eb065f0e5466ac23127fbd3fc6d1f2e4157a2d42a8d54c0bde943: Status 404 returned error can't find the container with id 0e2acabbb80eb065f0e5466ac23127fbd3fc6d1f2e4157a2d42a8d54c0bde943
Sep 17 20:48:58.610240 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:58.610197    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefccf4d3_5793_4300_ac5f_ea35e453da2f.slice/crio-7b566a55fdba62361d486d50264aa9e4ab6d42cbf30e646a81b541082900532c WatchSource:0}: Error finding container 7b566a55fdba62361d486d50264aa9e4ab6d42cbf30e646a81b541082900532c: Status 404 returned error can't find the container with id 7b566a55fdba62361d486d50264aa9e4ab6d42cbf30e646a81b541082900532c
Sep 17 20:48:58.633602 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:58.633581    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a25b3d5_790d_41a8_bea7_4689843c7d71.slice/crio-651d7391d9153566f156b6dd721134d1bf624449b9488ad02a5800cab817d283 WatchSource:0}: Error finding container 651d7391d9153566f156b6dd721134d1bf624449b9488ad02a5800cab817d283: Status 404 returned error can't find the container with id 651d7391d9153566f156b6dd721134d1bf624449b9488ad02a5800cab817d283
Sep 17 20:48:58.634481 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:58.634372    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d501871_a063_4d0a_9267_d4216673f3ef.slice/crio-39b8685d20cb4a31a3bb3e1317b0e59169660fbf951ba562ac7ac7510afb1d19 WatchSource:0}: Error finding container 39b8685d20cb4a31a3bb3e1317b0e59169660fbf951ba562ac7ac7510afb1d19: Status 404 returned error can't find the container with id 39b8685d20cb4a31a3bb3e1317b0e59169660fbf951ba562ac7ac7510afb1d19
Sep 17 20:48:58.635755 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:58.635656    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ddbfff5_cd86_477e_9152_7b0969bb0054.slice/crio-4c7d2ccb640164246a633b37210ce91750d4c9de82ea3b22c615be776da85e4d WatchSource:0}: Error finding container 4c7d2ccb640164246a633b37210ce91750d4c9de82ea3b22c615be776da85e4d: Status 404 returned error can't find the container with id 4c7d2ccb640164246a633b37210ce91750d4c9de82ea3b22c615be776da85e4d
Sep 17 20:48:58.637113 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:58.637083    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77f9970b_70e4_446e_954f_a37db50b183d.slice/crio-a95b2b84579ea6ccffda17165c32c14a175c835aa95e078dc3b6f822c55745cd WatchSource:0}: Error finding container a95b2b84579ea6ccffda17165c32c14a175c835aa95e078dc3b6f822c55745cd: Status 404 returned error can't find the container with id a95b2b84579ea6ccffda17165c32c14a175c835aa95e078dc3b6f822c55745cd
Sep 17 20:48:58.644277 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:48:58.644256    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda83fb9fa_5c05_488c_bfc6_5c8e15bd62cd.slice/crio-e3297cf644582eeee241fb867d997c85c75336abcdae18dea3e65ef5d737b372 WatchSource:0}: Error finding container e3297cf644582eeee241fb867d997c85c75336abcdae18dea3e65ef5d737b372: Status 404 returned error can't find the container with id e3297cf644582eeee241fb867d997c85c75336abcdae18dea3e65ef5d737b372
Sep 17 20:48:58.883051 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:58.883022    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d42a790e-18b7-4478-b0c7-3c7cc4372fff-metrics-certs\") pod \"network-metrics-daemon-8b6dt\" (UID: \"d42a790e-18b7-4478-b0c7-3c7cc4372fff\") " pod="openshift-multus/network-metrics-daemon-8b6dt"
Sep 17 20:48:58.883196 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:48:58.883178    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:48:58.883257 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:48:58.883247    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d42a790e-18b7-4478-b0c7-3c7cc4372fff-metrics-certs podName:d42a790e-18b7-4478-b0c7-3c7cc4372fff nodeName:}" failed. No retries permitted until 2026-09-17 20:49:00.883232801 +0000 UTC m=+6.035677157 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d42a790e-18b7-4478-b0c7-3c7cc4372fff-metrics-certs") pod "network-metrics-daemon-8b6dt" (UID: "d42a790e-18b7-4478-b0c7-3c7cc4372fff") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:48:58.983414 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:58.983383    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/334a13ac-ccb2-43eb-bd43-566981c058ff-original-pull-secret\") pod \"global-pull-secret-syncer-dpwhq\" (UID: \"334a13ac-ccb2-43eb-bd43-566981c058ff\") " pod="kube-system/global-pull-secret-syncer-dpwhq"
Sep 17 20:48:58.983547 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:58.983445    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqcxv\" (UniqueName: \"kubernetes.io/projected/53dee621-a806-480a-80c6-ff174deb8ed4-kube-api-access-fqcxv\") pod \"network-check-target-l28wp\" (UID: \"53dee621-a806-480a-80c6-ff174deb8ed4\") " pod="openshift-network-diagnostics/network-check-target-l28wp"
Sep 17 20:48:58.983603 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:48:58.983578    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 20:48:58.983672 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:48:58.983660    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/334a13ac-ccb2-43eb-bd43-566981c058ff-original-pull-secret podName:334a13ac-ccb2-43eb-bd43-566981c058ff nodeName:}" failed. No retries permitted until 2026-09-17 20:49:00.983642843 +0000 UTC m=+6.136087206 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/334a13ac-ccb2-43eb-bd43-566981c058ff-original-pull-secret") pod "global-pull-secret-syncer-dpwhq" (UID: "334a13ac-ccb2-43eb-bd43-566981c058ff") : object "kube-system"/"original-pull-secret" not registered
Sep 17 20:48:58.983718 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:48:58.983592    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 20:48:58.983718 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:48:58.983697    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 20:48:58.983718 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:48:58.983712    2575 projected.go:196] Error preparing data for projected volume kube-api-access-fqcxv for pod openshift-network-diagnostics/network-check-target-l28wp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:48:58.983809 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:48:58.983761    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/53dee621-a806-480a-80c6-ff174deb8ed4-kube-api-access-fqcxv podName:53dee621-a806-480a-80c6-ff174deb8ed4 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:00.983747613 +0000 UTC m=+6.136191988 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-fqcxv" (UniqueName: "kubernetes.io/projected/53dee621-a806-480a-80c6-ff174deb8ed4-kube-api-access-fqcxv") pod "network-check-target-l28wp" (UID: "53dee621-a806-480a-80c6-ff174deb8ed4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:48:59.307758 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:59.307693    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 20:43:57 +0000 UTC" deadline="2028-02-29 06:50:42.235828657 +0000 UTC"
Sep 17 20:48:59.307758 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:59.307756    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12706h1m42.92807719s"
Sep 17 20:48:59.351756 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:59.351505    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8b6dt"
Sep 17 20:48:59.351940 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:48:59.351905    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8b6dt" podUID="d42a790e-18b7-4478-b0c7-3c7cc4372fff"
Sep 17 20:48:59.352383 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:59.352361    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dpwhq"
Sep 17 20:48:59.352494 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:48:59.352471    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dpwhq" podUID="334a13ac-ccb2-43eb-bd43-566981c058ff"
Sep 17 20:48:59.352591 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:59.352577    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l28wp"
Sep 17 20:48:59.352683 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:48:59.352660    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l28wp" podUID="53dee621-a806-480a-80c6-ff174deb8ed4"
Sep 17 20:48:59.369946 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:59.369813    2575 generic.go:356] "Generic (PLEG): container finished" podID="92e7e66bfcc6d2b6f25c825e57da7ac2" containerID="8705e512defcf5b08cdc6f138102adfda0ce31c1e160bb3f5223db82e90eeeb6" exitCode=0
Sep 17 20:48:59.369946 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:59.369904    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-36.ec2.internal" event={"ID":"92e7e66bfcc6d2b6f25c825e57da7ac2","Type":"ContainerDied","Data":"8705e512defcf5b08cdc6f138102adfda0ce31c1e160bb3f5223db82e90eeeb6"}
Sep 17 20:48:59.372842 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:59.372815    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-r58z7" event={"ID":"a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd","Type":"ContainerStarted","Data":"e3297cf644582eeee241fb867d997c85c75336abcdae18dea3e65ef5d737b372"}
Sep 17 20:48:59.375906 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:59.375881    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-c4h4p" event={"ID":"77f9970b-70e4-446e-954f-a37db50b183d","Type":"ContainerStarted","Data":"a95b2b84579ea6ccffda17165c32c14a175c835aa95e078dc3b6f822c55745cd"}
Sep 17 20:48:59.377594 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:59.377569    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zcsbp" event={"ID":"5ddbfff5-cd86-477e-9152-7b0969bb0054","Type":"ContainerStarted","Data":"4c7d2ccb640164246a633b37210ce91750d4c9de82ea3b22c615be776da85e4d"}
Sep 17 20:48:59.379763 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:59.379671    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m" event={"ID":"7d501871-a063-4d0a-9267-d4216673f3ef","Type":"ContainerStarted","Data":"39b8685d20cb4a31a3bb3e1317b0e59169660fbf951ba562ac7ac7510afb1d19"}
Sep 17 20:48:59.381826 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:59.381763    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mk66j" event={"ID":"1a25b3d5-790d-41a8-bea7-4689843c7d71","Type":"ContainerStarted","Data":"651d7391d9153566f156b6dd721134d1bf624449b9488ad02a5800cab817d283"}
Sep 17 20:48:59.383398 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:59.383352    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cj2s8" event={"ID":"f3d70fb8-dd97-4d0e-99b1-694aa64834ca","Type":"ContainerStarted","Data":"0e2acabbb80eb065f0e5466ac23127fbd3fc6d1f2e4157a2d42a8d54c0bde943"}
Sep 17 20:48:59.386564 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:59.386536    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-48p9f" event={"ID":"5af35da6-3bf6-49ca-9a78-0218c7166c83","Type":"ContainerStarted","Data":"f6afa78fedacc0b60feea59b4a81b3d843103e32cd911b286d816467f6e0f9e9"}
Sep 17 20:48:59.389128 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:59.389101    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-36.ec2.internal" event={"ID":"21ea1fa7387769fb888cd296b1eefe37","Type":"ContainerStarted","Data":"f1779b8b24ec2f59e261aaeebbc42c093a9324180bced132b298d4532308d6ce"}
Sep 17 20:48:59.397985 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:59.397917    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rk9ss" event={"ID":"efccf4d3-5793-4300-ac5f-ea35e453da2f","Type":"ContainerStarted","Data":"7b566a55fdba62361d486d50264aa9e4ab6d42cbf30e646a81b541082900532c"}
Sep 17 20:48:59.402688 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:59.402114    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-36.ec2.internal" podStartSLOduration=2.402056861 podStartE2EDuration="2.402056861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:48:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:48:59.401749207 +0000 UTC m=+4.554193608" watchObservedRunningTime="2026-09-17 20:48:59.402056861 +0000 UTC m=+4.554501259"
Sep 17 20:48:59.402688 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:48:59.402676    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2lrgx" event={"ID":"0369efa0-b31a-4fa5-b426-95ba067ea731","Type":"ContainerStarted","Data":"f4124c6a6beb36cdd049e480875f097cef2f9a9bbd9f0d72d5dbf9f8f92a9df3"}
Sep 17 20:49:00.424605 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:00.424121    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-36.ec2.internal" event={"ID":"92e7e66bfcc6d2b6f25c825e57da7ac2","Type":"ContainerStarted","Data":"89df6e3713f3171281c1e7df9f11005920fc3c4003dd0df3a91fa18338fc01c9"}
Sep 17 20:49:00.438611 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:00.438547    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-36.ec2.internal" podStartSLOduration=3.438524096 podStartE2EDuration="3.438524096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:48:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:00.438060725 +0000 UTC m=+5.590505124" watchObservedRunningTime="2026-09-17 20:49:00.438524096 +0000 UTC m=+5.590968476"
Sep 17 20:49:00.899562 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:00.899523    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d42a790e-18b7-4478-b0c7-3c7cc4372fff-metrics-certs\") pod \"network-metrics-daemon-8b6dt\" (UID: \"d42a790e-18b7-4478-b0c7-3c7cc4372fff\") " pod="openshift-multus/network-metrics-daemon-8b6dt"
Sep 17 20:49:00.899771 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:00.899749    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:00.899843 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:00.899825    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d42a790e-18b7-4478-b0c7-3c7cc4372fff-metrics-certs podName:d42a790e-18b7-4478-b0c7-3c7cc4372fff nodeName:}" failed. No retries permitted until 2026-09-17 20:49:04.899805307 +0000 UTC m=+10.052249669 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d42a790e-18b7-4478-b0c7-3c7cc4372fff-metrics-certs") pod "network-metrics-daemon-8b6dt" (UID: "d42a790e-18b7-4478-b0c7-3c7cc4372fff") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:01.001862 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:01.000901    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/334a13ac-ccb2-43eb-bd43-566981c058ff-original-pull-secret\") pod \"global-pull-secret-syncer-dpwhq\" (UID: \"334a13ac-ccb2-43eb-bd43-566981c058ff\") " pod="kube-system/global-pull-secret-syncer-dpwhq"
Sep 17 20:49:01.001862 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:01.000993    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqcxv\" (UniqueName: \"kubernetes.io/projected/53dee621-a806-480a-80c6-ff174deb8ed4-kube-api-access-fqcxv\") pod \"network-check-target-l28wp\" (UID: \"53dee621-a806-480a-80c6-ff174deb8ed4\") " pod="openshift-network-diagnostics/network-check-target-l28wp"
Sep 17 20:49:01.001862 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:01.001144    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 20:49:01.001862 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:01.001166    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 20:49:01.001862 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:01.001181    2575 projected.go:196] Error preparing data for projected volume kube-api-access-fqcxv for pod openshift-network-diagnostics/network-check-target-l28wp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:01.001862 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:01.001252    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/53dee621-a806-480a-80c6-ff174deb8ed4-kube-api-access-fqcxv podName:53dee621-a806-480a-80c6-ff174deb8ed4 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:05.001232676 +0000 UTC m=+10.153677046 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-fqcxv" (UniqueName: "kubernetes.io/projected/53dee621-a806-480a-80c6-ff174deb8ed4-kube-api-access-fqcxv") pod "network-check-target-l28wp" (UID: "53dee621-a806-480a-80c6-ff174deb8ed4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:01.001862 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:01.001760    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:01.001862 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:01.001820    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/334a13ac-ccb2-43eb-bd43-566981c058ff-original-pull-secret podName:334a13ac-ccb2-43eb-bd43-566981c058ff nodeName:}" failed. No retries permitted until 2026-09-17 20:49:05.001802715 +0000 UTC m=+10.154247090 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/334a13ac-ccb2-43eb-bd43-566981c058ff-original-pull-secret") pod "global-pull-secret-syncer-dpwhq" (UID: "334a13ac-ccb2-43eb-bd43-566981c058ff") : object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:01.351231 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:01.351195    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l28wp"
Sep 17 20:49:01.351424 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:01.351337    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l28wp" podUID="53dee621-a806-480a-80c6-ff174deb8ed4"
Sep 17 20:49:01.351764 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:01.351744    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8b6dt"
Sep 17 20:49:01.351881 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:01.351854    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8b6dt" podUID="d42a790e-18b7-4478-b0c7-3c7cc4372fff"
Sep 17 20:49:01.354107 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:01.354080    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dpwhq"
Sep 17 20:49:01.354294 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:01.354248    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dpwhq" podUID="334a13ac-ccb2-43eb-bd43-566981c058ff"
Sep 17 20:49:03.351888 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:03.351172    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8b6dt"
Sep 17 20:49:03.351888 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:03.351330    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8b6dt" podUID="d42a790e-18b7-4478-b0c7-3c7cc4372fff"
Sep 17 20:49:03.351888 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:03.351759    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dpwhq"
Sep 17 20:49:03.352528 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:03.352503    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dpwhq" podUID="334a13ac-ccb2-43eb-bd43-566981c058ff"
Sep 17 20:49:03.352688 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:03.352671    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l28wp"
Sep 17 20:49:03.352788 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:03.352766    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l28wp" podUID="53dee621-a806-480a-80c6-ff174deb8ed4"
Sep 17 20:49:04.931716 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:04.931680    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d42a790e-18b7-4478-b0c7-3c7cc4372fff-metrics-certs\") pod \"network-metrics-daemon-8b6dt\" (UID: \"d42a790e-18b7-4478-b0c7-3c7cc4372fff\") " pod="openshift-multus/network-metrics-daemon-8b6dt"
Sep 17 20:49:04.932210 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:04.931832    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:04.932210 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:04.931916    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d42a790e-18b7-4478-b0c7-3c7cc4372fff-metrics-certs podName:d42a790e-18b7-4478-b0c7-3c7cc4372fff nodeName:}" failed. No retries permitted until 2026-09-17 20:49:12.931890677 +0000 UTC m=+18.084335048 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d42a790e-18b7-4478-b0c7-3c7cc4372fff-metrics-certs") pod "network-metrics-daemon-8b6dt" (UID: "d42a790e-18b7-4478-b0c7-3c7cc4372fff") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:05.032667 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:05.032620    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqcxv\" (UniqueName: \"kubernetes.io/projected/53dee621-a806-480a-80c6-ff174deb8ed4-kube-api-access-fqcxv\") pod \"network-check-target-l28wp\" (UID: \"53dee621-a806-480a-80c6-ff174deb8ed4\") " pod="openshift-network-diagnostics/network-check-target-l28wp"
Sep 17 20:49:05.032821 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:05.032698    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/334a13ac-ccb2-43eb-bd43-566981c058ff-original-pull-secret\") pod \"global-pull-secret-syncer-dpwhq\" (UID: \"334a13ac-ccb2-43eb-bd43-566981c058ff\") " pod="kube-system/global-pull-secret-syncer-dpwhq"
Sep 17 20:49:05.033353 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:05.032892    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 20:49:05.033353 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:05.032925    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 20:49:05.033353 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:05.032939    2575 projected.go:196] Error preparing data for projected volume kube-api-access-fqcxv for pod openshift-network-diagnostics/network-check-target-l28wp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:05.033353 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:05.032999    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/53dee621-a806-480a-80c6-ff174deb8ed4-kube-api-access-fqcxv podName:53dee621-a806-480a-80c6-ff174deb8ed4 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:13.03297892 +0000 UTC m=+18.185423288 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-fqcxv" (UniqueName: "kubernetes.io/projected/53dee621-a806-480a-80c6-ff174deb8ed4-kube-api-access-fqcxv") pod "network-check-target-l28wp" (UID: "53dee621-a806-480a-80c6-ff174deb8ed4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:05.033353 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:05.033101    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:05.033353 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:05.033165    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/334a13ac-ccb2-43eb-bd43-566981c058ff-original-pull-secret podName:334a13ac-ccb2-43eb-bd43-566981c058ff nodeName:}" failed. No retries permitted until 2026-09-17 20:49:13.033148153 +0000 UTC m=+18.185592517 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/334a13ac-ccb2-43eb-bd43-566981c058ff-original-pull-secret") pod "global-pull-secret-syncer-dpwhq" (UID: "334a13ac-ccb2-43eb-bd43-566981c058ff") : object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:05.351690 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:05.351657    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l28wp"
Sep 17 20:49:05.351892 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:05.351792    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l28wp" podUID="53dee621-a806-480a-80c6-ff174deb8ed4"
Sep 17 20:49:05.353096 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:05.353071    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8b6dt"
Sep 17 20:49:05.353211 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:05.353192    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8b6dt" podUID="d42a790e-18b7-4478-b0c7-3c7cc4372fff"
Sep 17 20:49:05.353286 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:05.353231    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dpwhq"
Sep 17 20:49:05.353367 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:05.353340    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dpwhq" podUID="334a13ac-ccb2-43eb-bd43-566981c058ff"
Sep 17 20:49:07.351399 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:07.351367    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l28wp"
Sep 17 20:49:07.351876 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:07.351364    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8b6dt"
Sep 17 20:49:07.351876 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:07.351494    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dpwhq"
Sep 17 20:49:07.351876 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:07.351508    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l28wp" podUID="53dee621-a806-480a-80c6-ff174deb8ed4"
Sep 17 20:49:07.351876 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:07.351604    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dpwhq" podUID="334a13ac-ccb2-43eb-bd43-566981c058ff"
Sep 17 20:49:07.351876 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:07.351699    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8b6dt" podUID="d42a790e-18b7-4478-b0c7-3c7cc4372fff"
Sep 17 20:49:09.351536 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:09.351502    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8b6dt"
Sep 17 20:49:09.351951 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:09.351492    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l28wp"
Sep 17 20:49:09.351951 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:09.351502    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dpwhq"
Sep 17 20:49:09.351951 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:09.351671    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8b6dt" podUID="d42a790e-18b7-4478-b0c7-3c7cc4372fff"
Sep 17 20:49:09.351951 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:09.351729    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dpwhq" podUID="334a13ac-ccb2-43eb-bd43-566981c058ff"
Sep 17 20:49:09.351951 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:09.351818    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l28wp" podUID="53dee621-a806-480a-80c6-ff174deb8ed4"
Sep 17 20:49:11.351151 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:11.351112    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l28wp"
Sep 17 20:49:11.351583 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:11.351112    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dpwhq"
Sep 17 20:49:11.351583 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:11.351238    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l28wp" podUID="53dee621-a806-480a-80c6-ff174deb8ed4"
Sep 17 20:49:11.351583 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:11.351113    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8b6dt"
Sep 17 20:49:11.351583 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:11.351294    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dpwhq" podUID="334a13ac-ccb2-43eb-bd43-566981c058ff"
Sep 17 20:49:11.351583 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:11.351411    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8b6dt" podUID="d42a790e-18b7-4478-b0c7-3c7cc4372fff"
Sep 17 20:49:12.998135 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:12.998100    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d42a790e-18b7-4478-b0c7-3c7cc4372fff-metrics-certs\") pod \"network-metrics-daemon-8b6dt\" (UID: \"d42a790e-18b7-4478-b0c7-3c7cc4372fff\") " pod="openshift-multus/network-metrics-daemon-8b6dt"
Sep 17 20:49:12.998603 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:12.998228    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:12.998603 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:12.998287    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d42a790e-18b7-4478-b0c7-3c7cc4372fff-metrics-certs podName:d42a790e-18b7-4478-b0c7-3c7cc4372fff nodeName:}" failed. No retries permitted until 2026-09-17 20:49:28.998270598 +0000 UTC m=+34.150714957 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d42a790e-18b7-4478-b0c7-3c7cc4372fff-metrics-certs") pod "network-metrics-daemon-8b6dt" (UID: "d42a790e-18b7-4478-b0c7-3c7cc4372fff") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:13.099138 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:13.099093    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqcxv\" (UniqueName: \"kubernetes.io/projected/53dee621-a806-480a-80c6-ff174deb8ed4-kube-api-access-fqcxv\") pod \"network-check-target-l28wp\" (UID: \"53dee621-a806-480a-80c6-ff174deb8ed4\") " pod="openshift-network-diagnostics/network-check-target-l28wp"
Sep 17 20:49:13.099337 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:13.099151    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/334a13ac-ccb2-43eb-bd43-566981c058ff-original-pull-secret\") pod \"global-pull-secret-syncer-dpwhq\" (UID: \"334a13ac-ccb2-43eb-bd43-566981c058ff\") " pod="kube-system/global-pull-secret-syncer-dpwhq"
Sep 17 20:49:13.099337 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:13.099231    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 20:49:13.099337 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:13.099246    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:13.099337 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:13.099260    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 20:49:13.099337 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:13.099273    2575 projected.go:196] Error preparing data for projected volume kube-api-access-fqcxv for pod openshift-network-diagnostics/network-check-target-l28wp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:13.099337 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:13.099297    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/334a13ac-ccb2-43eb-bd43-566981c058ff-original-pull-secret podName:334a13ac-ccb2-43eb-bd43-566981c058ff nodeName:}" failed. No retries permitted until 2026-09-17 20:49:29.099284943 +0000 UTC m=+34.251729298 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/334a13ac-ccb2-43eb-bd43-566981c058ff-original-pull-secret") pod "global-pull-secret-syncer-dpwhq" (UID: "334a13ac-ccb2-43eb-bd43-566981c058ff") : object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:13.099337 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:13.099321    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/53dee621-a806-480a-80c6-ff174deb8ed4-kube-api-access-fqcxv podName:53dee621-a806-480a-80c6-ff174deb8ed4 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:29.099306394 +0000 UTC m=+34.251750748 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-fqcxv" (UniqueName: "kubernetes.io/projected/53dee621-a806-480a-80c6-ff174deb8ed4-kube-api-access-fqcxv") pod "network-check-target-l28wp" (UID: "53dee621-a806-480a-80c6-ff174deb8ed4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:13.351192 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:13.351157    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l28wp"
Sep 17 20:49:13.351392 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:13.351157    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dpwhq"
Sep 17 20:49:13.351392 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:13.351297    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8b6dt"
Sep 17 20:49:13.351392 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:13.351293    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l28wp" podUID="53dee621-a806-480a-80c6-ff174deb8ed4"
Sep 17 20:49:13.351527 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:13.351406    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dpwhq" podUID="334a13ac-ccb2-43eb-bd43-566981c058ff"
Sep 17 20:49:13.351578 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:13.351512    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8b6dt" podUID="d42a790e-18b7-4478-b0c7-3c7cc4372fff"
Sep 17 20:49:15.352136 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:15.352069    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l28wp"
Sep 17 20:49:15.352576 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:15.352180    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l28wp" podUID="53dee621-a806-480a-80c6-ff174deb8ed4"
Sep 17 20:49:15.352576 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:15.352269    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8b6dt"
Sep 17 20:49:15.352576 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:15.352395    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dpwhq"
Sep 17 20:49:15.352576 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:15.352416    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8b6dt" podUID="d42a790e-18b7-4478-b0c7-3c7cc4372fff"
Sep 17 20:49:15.352576 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:15.352471    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dpwhq" podUID="334a13ac-ccb2-43eb-bd43-566981c058ff"
Sep 17 20:49:17.353901 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:17.351036    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8b6dt"
Sep 17 20:49:17.353901 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:17.351037    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dpwhq"
Sep 17 20:49:17.353901 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:17.351071    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l28wp"
Sep 17 20:49:17.353901 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:17.351575    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8b6dt" podUID="d42a790e-18b7-4478-b0c7-3c7cc4372fff"
Sep 17 20:49:17.353901 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:17.351879    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l28wp" podUID="53dee621-a806-480a-80c6-ff174deb8ed4"
Sep 17 20:49:17.353901 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:17.352026    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dpwhq" podUID="334a13ac-ccb2-43eb-bd43-566981c058ff"
Sep 17 20:49:17.471706 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:17.471666    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m" event={"ID":"7d501871-a063-4d0a-9267-d4216673f3ef","Type":"ContainerStarted","Data":"c56b09a03f671bee92f542ccd360164994577df7fd71fe2039ff236611b81928"}
Sep 17 20:49:17.471911 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:17.471771    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m" event={"ID":"7d501871-a063-4d0a-9267-d4216673f3ef","Type":"ContainerStarted","Data":"873199da914f37d2b6c77c38a156677c834d162292435e091803e2394a1ff513"}
Sep 17 20:49:17.471911 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:17.471816    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m" event={"ID":"7d501871-a063-4d0a-9267-d4216673f3ef","Type":"ContainerStarted","Data":"9182a289eb14c1b725ad98dc4ad2bf3a89415034d685358ba02027af12cc4be1"}
Sep 17 20:49:17.471911 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:17.471866    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m" event={"ID":"7d501871-a063-4d0a-9267-d4216673f3ef","Type":"ContainerStarted","Data":"376f11934d0d90803cda777a8b6cda293981b4398a766e7c2b6b1757338e782a"}
Sep 17 20:49:17.471911 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:17.471898    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m" event={"ID":"7d501871-a063-4d0a-9267-d4216673f3ef","Type":"ContainerStarted","Data":"ccfc3185f9648499d62dad60c05b6d25e1cb2026242cc7b341563720f72fea94"}
Sep 17 20:49:17.472085 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:17.471940    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m" event={"ID":"7d501871-a063-4d0a-9267-d4216673f3ef","Type":"ContainerStarted","Data":"36254fe6ca6bc4cfb67ad31900f48b97c435143dd2d77e5ffa9fa3c97669f611"}
Sep 17 20:49:17.473400 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:17.473371    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mk66j" event={"ID":"1a25b3d5-790d-41a8-bea7-4689843c7d71","Type":"ContainerStarted","Data":"d648894adb944a5b3ffea0876592d9492f37b675c208d9d4c652df54f896b026"}
Sep 17 20:49:17.475098 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:17.475073    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cj2s8" event={"ID":"f3d70fb8-dd97-4d0e-99b1-694aa64834ca","Type":"ContainerStarted","Data":"b04c3b2f64a80562205f959934b2f9503e19046e8c5c69ae2fedb1da67ff95c8"}
Sep 17 20:49:17.476971 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:17.476945    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-48p9f" event={"ID":"5af35da6-3bf6-49ca-9a78-0218c7166c83","Type":"ContainerStarted","Data":"652662d275d8ee5ffd86aefc80aa30f23bd6abb64eefbe3e38d6356fd26f259b"}
Sep 17 20:49:17.478671 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:17.478619    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rk9ss" event={"ID":"efccf4d3-5793-4300-ac5f-ea35e453da2f","Type":"ContainerStarted","Data":"b7478d35a293f365cbfec02f8d27792d198e4be5c0a656c29e542ff5258c33f8"}
Sep 17 20:49:17.480361 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:17.480310    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2lrgx" event={"ID":"0369efa0-b31a-4fa5-b426-95ba067ea731","Type":"ContainerStarted","Data":"3136f8fc908462d63180ae1329c2e6e1894d3853b0a7457a0c0a6c78a9da0576"}
Sep 17 20:49:17.482079 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:17.482057    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-r58z7" event={"ID":"a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd","Type":"ContainerStarted","Data":"86dbf8e44870a5f1b1951826030686a5e1f54fe959f4d11f419543740dc80df8"}
Sep 17 20:49:17.483763 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:17.483737    2575 generic.go:356] "Generic (PLEG): container finished" podID="5ddbfff5-cd86-477e-9152-7b0969bb0054" containerID="9db0ea29be88af7181956de9b7f7c34028a03e41420b500a1d802f658326a343" exitCode=0
Sep 17 20:49:17.483870 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:17.483774    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zcsbp" event={"ID":"5ddbfff5-cd86-477e-9152-7b0969bb0054","Type":"ContainerDied","Data":"9db0ea29be88af7181956de9b7f7c34028a03e41420b500a1d802f658326a343"}
Sep 17 20:49:17.532956 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:17.532706    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-mk66j" podStartSLOduration=7.242228242 podStartE2EDuration="21.532685302s" totalImagesPullingTime="14.29045706s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:48:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:17.506073301 +0000 UTC m=+22.658517678" watchObservedRunningTime="2026-09-17 20:49:17.532685302 +0000 UTC m=+22.685129693"
Sep 17 20:49:17.555072 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:17.555040    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-mk66j"
Sep 17 20:49:17.555632 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:17.555609    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-mk66j"
Sep 17 20:49:17.566714 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:17.566669    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2lrgx" podStartSLOduration=3.687851608 podStartE2EDuration="21.566654508s" totalImagesPullingTime="17.8788029s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:48:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:17.547960882 +0000 UTC m=+22.700405259" watchObservedRunningTime="2026-09-17 20:49:17.566654508 +0000 UTC m=+22.719098878"
Sep 17 20:49:17.567154 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:17.567103    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-r58z7" podStartSLOduration=4.668415475 podStartE2EDuration="22.567095863s" totalImagesPullingTime="17.898680388s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:48:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:17.566791769 +0000 UTC m=+22.719236144" watchObservedRunningTime="2026-09-17 20:49:17.567095863 +0000 UTC m=+22.719540239"
Sep 17 20:49:17.583056 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:17.583013    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rk9ss" podStartSLOduration=4.712431839 podStartE2EDuration="22.582999078s" totalImagesPullingTime="17.870567239s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:48:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:17.582818682 +0000 UTC m=+22.735263058" watchObservedRunningTime="2026-09-17 20:49:17.582999078 +0000 UTC m=+22.735443473"
Sep 17 20:49:17.595463 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:17.595421    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-48p9f" podStartSLOduration=3.713005471 podStartE2EDuration="21.595382385s" totalImagesPullingTime="17.882376914s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:48:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:17.594752942 +0000 UTC m=+22.747197345" watchObservedRunningTime="2026-09-17 20:49:17.595382385 +0000 UTC m=+22.747826761"
Sep 17 20:49:18.110827 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:18.110801    2575 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 20:49:18.330996 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:18.330836    2575 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T20:49:18.110819624Z","UUID":"2875bef6-61e4-4e4b-adb9-02b2b0dd9083","Handler":null,"Name":"","Endpoint":""}
Sep 17 20:49:18.332368 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:18.332350    2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 20:49:18.332461 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:18.332380    2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 20:49:18.487745 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:18.487714    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-c4h4p" event={"ID":"77f9970b-70e4-446e-954f-a37db50b183d","Type":"ContainerStarted","Data":"aa2b68c992456d69fe474865aac0aa0854166c18629c7aafaf36235c82e2ce9c"}
Sep 17 20:49:18.489606 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:18.489579    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cj2s8" event={"ID":"f3d70fb8-dd97-4d0e-99b1-694aa64834ca","Type":"ContainerStarted","Data":"3cb6941e4b0faea3ea007caaa297b59be793ff7be5e712da326dd7c99d774861"}
Sep 17 20:49:18.518066 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:18.518000    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-c4h4p" podStartSLOduration=4.665583149 podStartE2EDuration="22.517977215s" totalImagesPullingTime="17.852394066s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:48:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:18.517779692 +0000 UTC m=+23.670224067" watchObservedRunningTime="2026-09-17 20:49:18.517977215 +0000 UTC m=+23.670421630"
Sep 17 20:49:19.351399 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:19.351364    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l28wp"
Sep 17 20:49:19.351582 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:19.351361    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8b6dt"
Sep 17 20:49:19.351582 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:19.351496    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l28wp" podUID="53dee621-a806-480a-80c6-ff174deb8ed4"
Sep 17 20:49:19.351694 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:19.351598    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8b6dt" podUID="d42a790e-18b7-4478-b0c7-3c7cc4372fff"
Sep 17 20:49:19.351694 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:19.351359    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dpwhq"
Sep 17 20:49:19.351792 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:19.351701    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dpwhq" podUID="334a13ac-ccb2-43eb-bd43-566981c058ff"
Sep 17 20:49:19.495720 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:19.495683    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m" event={"ID":"7d501871-a063-4d0a-9267-d4216673f3ef","Type":"ContainerStarted","Data":"aa8d387462fd4f172065f0ab7dec64f5c0b5cd8ab107065b2ece67979cd5e388"}
Sep 17 20:49:19.499090 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:19.499054    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cj2s8" event={"ID":"f3d70fb8-dd97-4d0e-99b1-694aa64834ca","Type":"ContainerStarted","Data":"294ebbb7bf2d5b7c767ff7ac76d034ee93bd4c71b40a71f45d34f1d79d76e03f"}
Sep 17 20:49:19.499227 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:19.499095    2575 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 20:49:19.518698 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:19.518655    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cj2s8" podStartSLOduration=3.398828915 podStartE2EDuration="23.518637194s" totalImagesPullingTime="20.119808279s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:48:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:19.518545833 +0000 UTC m=+24.670990210" watchObservedRunningTime="2026-09-17 20:49:19.518637194 +0000 UTC m=+24.671081573"
Sep 17 20:49:21.351635 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:21.351458    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l28wp"
Sep 17 20:49:21.352099 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:21.351470    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8b6dt"
Sep 17 20:49:21.352099 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:21.351720    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l28wp" podUID="53dee621-a806-480a-80c6-ff174deb8ed4"
Sep 17 20:49:21.352099 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:21.351822    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8b6dt" podUID="d42a790e-18b7-4478-b0c7-3c7cc4372fff"
Sep 17 20:49:21.352099 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:21.351501    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dpwhq"
Sep 17 20:49:21.352099 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:21.351997    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dpwhq" podUID="334a13ac-ccb2-43eb-bd43-566981c058ff"
Sep 17 20:49:22.509932 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:22.509889    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zcsbp" event={"ID":"5ddbfff5-cd86-477e-9152-7b0969bb0054","Type":"ContainerStarted","Data":"af09bcbe6c36baa1654b1e4c1d6497cb1b061f2843bbb9da499b3e7d8b8c3ab1"}
Sep 17 20:49:22.514457 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:22.514417    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m" event={"ID":"7d501871-a063-4d0a-9267-d4216673f3ef","Type":"ContainerStarted","Data":"41465c24e61b605f8d8642ccf566ce3e046b491ce2dbadff92bf08ce85f765ac"}
Sep 17 20:49:22.514726 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:22.514651    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m"
Sep 17 20:49:22.529944 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:22.529912    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m"
Sep 17 20:49:23.351603 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:23.351432    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dpwhq"
Sep 17 20:49:23.351741 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:23.351476    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l28wp"
Sep 17 20:49:23.351741 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:23.351687    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dpwhq" podUID="334a13ac-ccb2-43eb-bd43-566981c058ff"
Sep 17 20:49:23.351821 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:23.351490    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8b6dt"
Sep 17 20:49:23.351821 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:23.351759    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l28wp" podUID="53dee621-a806-480a-80c6-ff174deb8ed4"
Sep 17 20:49:23.351903 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:23.351830    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8b6dt" podUID="d42a790e-18b7-4478-b0c7-3c7cc4372fff"
Sep 17 20:49:23.518045 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:23.518016    2575 generic.go:356] "Generic (PLEG): container finished" podID="5ddbfff5-cd86-477e-9152-7b0969bb0054" containerID="af09bcbe6c36baa1654b1e4c1d6497cb1b061f2843bbb9da499b3e7d8b8c3ab1" exitCode=0
Sep 17 20:49:23.518444 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:23.518100    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zcsbp" event={"ID":"5ddbfff5-cd86-477e-9152-7b0969bb0054","Type":"ContainerDied","Data":"af09bcbe6c36baa1654b1e4c1d6497cb1b061f2843bbb9da499b3e7d8b8c3ab1"}
Sep 17 20:49:23.518644 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:23.518624    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m"
Sep 17 20:49:23.518727 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:23.518701    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m"
Sep 17 20:49:23.534652 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:23.534630    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m"
Sep 17 20:49:23.540362 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:23.540326    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m" podStartSLOduration=9.636684966 podStartE2EDuration="27.540312589s" totalImagesPullingTime="17.903627623s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:48:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:22.592047258 +0000 UTC m=+27.744491633" watchObservedRunningTime="2026-09-17 20:49:23.540312589 +0000 UTC m=+28.692756966"
Sep 17 20:49:24.397612 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:24.397531    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dpwhq"]
Sep 17 20:49:24.397741 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:24.397694    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dpwhq"
Sep 17 20:49:24.397897 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:24.397835    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dpwhq" podUID="334a13ac-ccb2-43eb-bd43-566981c058ff"
Sep 17 20:49:24.402483 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:24.402460    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-l28wp"]
Sep 17 20:49:24.402598 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:24.402584    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l28wp"
Sep 17 20:49:24.402702 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:24.402676    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l28wp" podUID="53dee621-a806-480a-80c6-ff174deb8ed4"
Sep 17 20:49:24.404098 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:24.404077    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8b6dt"]
Sep 17 20:49:24.404179 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:24.404159    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8b6dt"
Sep 17 20:49:24.404242 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:24.404226    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8b6dt" podUID="d42a790e-18b7-4478-b0c7-3c7cc4372fff"
Sep 17 20:49:24.522226 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:24.522188    2575 generic.go:356] "Generic (PLEG): container finished" podID="5ddbfff5-cd86-477e-9152-7b0969bb0054" containerID="d4a93f0b4bb98c9c16330d19fc480b2abe774d1b452e088a92883ef54ff9b053" exitCode=0
Sep 17 20:49:24.522605 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:24.522278    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zcsbp" event={"ID":"5ddbfff5-cd86-477e-9152-7b0969bb0054","Type":"ContainerDied","Data":"d4a93f0b4bb98c9c16330d19fc480b2abe774d1b452e088a92883ef54ff9b053"}
Sep 17 20:49:25.527039 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:25.527007    2575 generic.go:356] "Generic (PLEG): container finished" podID="5ddbfff5-cd86-477e-9152-7b0969bb0054" containerID="1d19de629951616c2cf4372345ec6ac29bff66e792609dd51037644d76333591" exitCode=0
Sep 17 20:49:25.527435 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:25.527084    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zcsbp" event={"ID":"5ddbfff5-cd86-477e-9152-7b0969bb0054","Type":"ContainerDied","Data":"1d19de629951616c2cf4372345ec6ac29bff66e792609dd51037644d76333591"}
Sep 17 20:49:26.351482 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:26.351392    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dpwhq"
Sep 17 20:49:26.351649 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:26.351405    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l28wp"
Sep 17 20:49:26.351649 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:26.351515    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8b6dt"
Sep 17 20:49:26.351649 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:26.351514    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dpwhq" podUID="334a13ac-ccb2-43eb-bd43-566981c058ff"
Sep 17 20:49:26.351649 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:26.351578    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l28wp" podUID="53dee621-a806-480a-80c6-ff174deb8ed4"
Sep 17 20:49:26.351822 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:26.351692    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8b6dt" podUID="d42a790e-18b7-4478-b0c7-3c7cc4372fff"
Sep 17 20:49:27.448042 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:27.447875    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-mk66j"
Sep 17 20:49:27.448462 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:27.448151    2575 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 20:49:27.448574 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:27.448556    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mk66j"
Sep 17 20:49:28.350672 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:28.350608    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dpwhq"
Sep 17 20:49:28.350891 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:28.350737    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l28wp"
Sep 17 20:49:28.350891 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:28.350761    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8b6dt"
Sep 17 20:49:28.350891 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:28.350754    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dpwhq" podUID="334a13ac-ccb2-43eb-bd43-566981c058ff"
Sep 17 20:49:28.350891 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:28.350865    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l28wp" podUID="53dee621-a806-480a-80c6-ff174deb8ed4"
Sep 17 20:49:28.351132 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:28.350934    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8b6dt" podUID="d42a790e-18b7-4478-b0c7-3c7cc4372fff"
Sep 17 20:49:29.016768 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:29.016738    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d42a790e-18b7-4478-b0c7-3c7cc4372fff-metrics-certs\") pod \"network-metrics-daemon-8b6dt\" (UID: \"d42a790e-18b7-4478-b0c7-3c7cc4372fff\") " pod="openshift-multus/network-metrics-daemon-8b6dt"
Sep 17 20:49:29.017150 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:29.016888    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:29.017150 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:29.016965    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d42a790e-18b7-4478-b0c7-3c7cc4372fff-metrics-certs podName:d42a790e-18b7-4478-b0c7-3c7cc4372fff nodeName:}" failed. No retries permitted until 2026-09-17 20:50:01.01694439 +0000 UTC m=+66.169388765 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d42a790e-18b7-4478-b0c7-3c7cc4372fff-metrics-certs") pod "network-metrics-daemon-8b6dt" (UID: "d42a790e-18b7-4478-b0c7-3c7cc4372fff") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:29.117085 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:29.117044    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/334a13ac-ccb2-43eb-bd43-566981c058ff-original-pull-secret\") pod \"global-pull-secret-syncer-dpwhq\" (UID: \"334a13ac-ccb2-43eb-bd43-566981c058ff\") " pod="kube-system/global-pull-secret-syncer-dpwhq"
Sep 17 20:49:29.117305 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:29.117212    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqcxv\" (UniqueName: \"kubernetes.io/projected/53dee621-a806-480a-80c6-ff174deb8ed4-kube-api-access-fqcxv\") pod \"network-check-target-l28wp\" (UID: \"53dee621-a806-480a-80c6-ff174deb8ed4\") " pod="openshift-network-diagnostics/network-check-target-l28wp"
Sep 17 20:49:29.117305 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:29.117217    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:29.117305 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:29.117301    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 20:49:29.117475 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:29.117319    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 20:49:29.117475 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:29.117332    2575 projected.go:196] Error preparing data for projected volume kube-api-access-fqcxv for pod openshift-network-diagnostics/network-check-target-l28wp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:29.117475 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:29.117333    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/334a13ac-ccb2-43eb-bd43-566981c058ff-original-pull-secret podName:334a13ac-ccb2-43eb-bd43-566981c058ff nodeName:}" failed. No retries permitted until 2026-09-17 20:50:01.11731802 +0000 UTC m=+66.269762374 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/334a13ac-ccb2-43eb-bd43-566981c058ff-original-pull-secret") pod "global-pull-secret-syncer-dpwhq" (UID: "334a13ac-ccb2-43eb-bd43-566981c058ff") : object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:29.117475 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:29.117395    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/53dee621-a806-480a-80c6-ff174deb8ed4-kube-api-access-fqcxv podName:53dee621-a806-480a-80c6-ff174deb8ed4 nodeName:}" failed. No retries permitted until 2026-09-17 20:50:01.117376885 +0000 UTC m=+66.269821239 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-fqcxv" (UniqueName: "kubernetes.io/projected/53dee621-a806-480a-80c6-ff174deb8ed4-kube-api-access-fqcxv") pod "network-check-target-l28wp" (UID: "53dee621-a806-480a-80c6-ff174deb8ed4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:29.158517 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:29.158492    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-36.ec2.internal" event="NodeReady"
Sep 17 20:49:29.158637 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:29.158601    2575 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 20:49:29.193892 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:29.193864    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6c4b45cb74-s4gzt"]
Sep 17 20:49:29.240818 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:29.240793    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6c4b45cb74-s4gzt"]
Sep 17 20:49:29.240958 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:29.240829    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-z4lgm"]
Sep 17 20:49:29.240958 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:29.240922    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c4b45cb74-s4gzt"
Sep 17 20:49:29.243227 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:29.243208    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 20:49:29.243339 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:29.243255    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 20:49:29.243339 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:29.243279    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 20:49:29.243511 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:29.243353    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-wszs2\""
Sep 17 20:49:29.249036 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:29.248667    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 20:49:29.264479 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:29.264460    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-d7m9q"]
Sep 17 20:49:29.264585 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:29.264571    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-z4lgm"
Sep 17 20:49:29.266679 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:29.266628    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-b5xsj\""
Sep 17 20:49:29.266911 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:29.266895    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 20:49:29.266986 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:29.266915    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 20:49:29.266986 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:29.266962    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 20:49:29.284874 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:29.284837    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-z4lgm"]
Sep 17 20:49:29.284953 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:29.284882    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-d7m9q"]
Sep 17 20:49:29.284991 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:29.284964    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-d7m9q"
Sep 17 20:49:29.287122 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:29.287108    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 20:49:29.287254 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:29.287238    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 20:49:29.287307 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:29.287278    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2jjw4\""
Sep 17 20:49:29.318450 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:29.318427    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9dda82c0-e203-412d-8d70-bffaf6285281-registry-tls\") pod \"image-registry-6c4b45cb74-s4gzt\" (UID: \"9dda82c0-e203-412d-8d70-bffaf6285281\") " pod="openshift-image-registry/image-registry-6c4b45cb74-s4gzt"
Sep 17 20:49:29.318584 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:29.318489    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9dda82c0-e203-412d-8d70-bffaf6285281-trusted-ca\") pod \"image-registry-6c4b45cb74-s4gzt\" (UID: \"9dda82c0-e203-412d-8d70-bffaf6285281\") " pod="openshift-image-registry/image-registry-6c4b45cb74-s4gzt"
Sep 17 20:49:29.318584 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:29.318514    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm9qs\" (UniqueName: \"kubernetes.io/projected/9dda82c0-e203-412d-8d70-bffaf6285281-kube-api-access-bm9qs\") pod \"image-registry-6c4b45cb74-s4gzt\" (UID: \"9dda82c0-e203-412d-8d70-bffaf6285281\") " pod="openshift-image-registry/image-registry-6c4b45cb74-s4gzt"
Sep 17 20:49:29.318584 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:29.318540    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9dda82c0-e203-412d-8d70-bffaf6285281-image-registry-private-configuration\") pod \"image-registry-6c4b45cb74-s4gzt\" (UID: \"9dda82c0-e203-412d-8d70-bffaf6285281\") " pod="openshift-image-registry/image-registry-6c4b45cb74-s4gzt"
Sep 17 20:49:29.318584 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:29.318555    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9dda82c0-e203-412d-8d70-bffaf6285281-registry-certificates\") pod \"image-registry-6c4b45cb74-s4gzt\" (UID: \"9dda82c0-e203-412d-8d70-bffaf6285281\") " pod="openshift-image-registry/image-registry-6c4b45cb74-s4gzt"
Sep 17 20:49:29.318584 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:29.318572    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9dda82c0-e203-412d-8d70-bffaf6285281-installation-pull-secrets\") pod \"image-registry-6c4b45cb74-s4gzt\" (UID: \"9dda82c0-e203-412d-8d70-bffaf6285281\") " pod="openshift-image-registry/image-registry-6c4b45cb74-s4gzt"
Sep 17 20:49:29.318768 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:29.318653    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9dda82c0-e203-412d-8d70-bffaf6285281-bound-sa-token\") pod \"image-registry-6c4b45cb74-s4gzt\" (UID: \"9dda82c0-e203-412d-8d70-bffaf6285281\") " pod="openshift-image-registry/image-registry-6c4b45cb74-s4gzt"
Sep 17 20:49:29.318768 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:29.318676    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9dda82c0-e203-412d-8d70-bffaf6285281-ca-trust-extracted\") pod \"image-registry-6c4b45cb74-s4gzt\" (UID: \"9dda82c0-e203-412d-8d70-bffaf6285281\") " pod="openshift-image-registry/image-registry-6c4b45cb74-s4gzt"
Sep 17 20:49:29.373114 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:29.373070    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:49:29.420063 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:29.420031    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9dda82c0-e203-412d-8d70-bffaf6285281-trusted-ca\") pod \"image-registry-6c4b45cb74-s4gzt\" (UID: \"9dda82c0-e203-412d-8d70-bffaf6285281\") " pod="openshift-image-registry/image-registry-6c4b45cb74-s4gzt"
Sep 17 20:49:29.420224 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:29.420093    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm9qs\" (UniqueName: \"kubernetes.io/projected/9dda82c0-e203-412d-8d70-bffaf6285281-kube-api-access-bm9qs\") pod \"image-registry-6c4b45cb74-s4gzt\" (UID: \"9dda82c0-e203-412d-8d70-bffaf6285281\") " pod="openshift-image-registry/image-registry-6c4b45cb74-s4gzt"
Sep 17 20:49:29.420224 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:29.420143    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9dda82c0-e203-412d-8d70-bffaf6285281-image-registry-private-configuration\") pod \"image-registry-6c4b45cb74-s4gzt\" (UID: \"9dda82c0-e203-412d-8d70-bffaf6285281\") " pod="openshift-image-registry/image-registry-6c4b45cb74-s4gzt"
Sep 17 20:49:29.420224 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:29.420172    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9dda82c0-e203-412d-8d70-bffaf6285281-registry-certificates\") pod \"image-registry-6c4b45cb74-s4gzt\" (UID: \"9dda82c0-e203-412d-8d70-bffaf6285281\") " pod="openshift-image-registry/image-registry-6c4b45cb74-s4gzt"
Sep 17 20:49:29.420224 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:29.420196    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9dda82c0-e203-412d-8d70-bffaf6285281-installation-pull-secrets\") pod \"image-registry-6c4b45cb74-s4gzt\" (UID: \"9dda82c0-e203-412d-8d70-bffaf6285281\") " pod="openshift-image-registry/image-registry-6c4b45cb74-s4gzt"
Sep 17 20:49:29.420441 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:29.420240    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9dda82c0-e203-412d-8d70-bffaf6285281-bound-sa-token\") pod \"image-registry-6c4b45cb74-s4gzt\" (UID: \"9dda82c0-e203-412d-8d70-bffaf6285281\") " pod="openshift-image-registry/image-registry-6c4b45cb74-s4gzt"
Sep 17 20:49:29.420528 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:29.420507    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kn79\" (UniqueName: \"kubernetes.io/projected/3afbe98f-5052-44e2-9d4b-da4bd2d30c3a-kube-api-access-4kn79\") pod \"dns-default-d7m9q\" (UID: \"3afbe98f-5052-44e2-9d4b-da4bd2d30c3a\") " pod="openshift-dns/dns-default-d7m9q"
Sep 17 20:49:29.420586 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:29.420548    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9dda82c0-e203-412d-8d70-bffaf6285281-ca-trust-extracted\") pod \"image-registry-6c4b45cb74-s4gzt\" (UID: \"9dda82c0-e203-412d-8d70-bffaf6285281\") " pod="openshift-image-registry/image-registry-6c4b45cb74-s4gzt"
Sep 17 20:49:29.420586 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:29.420575    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54ba4bcf-1779-4a9d-a5c3-d3bc4507466c-cert\") pod \"ingress-canary-z4lgm\" (UID: \"54ba4bcf-1779-4a9d-a5c3-d3bc4507466c\") " pod="openshift-ingress-canary/ingress-canary-z4lgm"
Sep 17 20:49:29.420678 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:29.420607    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwgbm\" (UniqueName: \"kubernetes.io/projected/54ba4bcf-1779-4a9d-a5c3-d3bc4507466c-kube-api-access-lwgbm\") pod \"ingress-canary-z4lgm\" (UID: \"54ba4bcf-1779-4a9d-a5c3-d3bc4507466c\") " pod="openshift-ingress-canary/ingress-canary-z4lgm"
Sep 17 20:49:29.420678 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:29.420639    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9dda82c0-e203-412d-8d70-bffaf6285281-registry-tls\") pod \"image-registry-6c4b45cb74-s4gzt\" (UID: \"9dda82c0-e203-412d-8d70-bffaf6285281\") " pod="openshift-image-registry/image-registry-6c4b45cb74-s4gzt"
Sep 17 20:49:29.420678 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:29.420662    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3afbe98f-5052-44e2-9d4b-da4bd2d30c3a-config-volume\") pod \"dns-default-d7m9q\" (UID: \"3afbe98f-5052-44e2-9d4b-da4bd2d30c3a\") " pod="openshift-dns/dns-default-d7m9q"
Sep 17 20:49:29.420814 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:29.420689    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3afbe98f-5052-44e2-9d4b-da4bd2d30c3a-metrics-tls\") pod \"dns-default-d7m9q\" (UID: \"3afbe98f-5052-44e2-9d4b-da4bd2d30c3a\") " pod="openshift-dns/dns-default-d7m9q"
Sep 17 20:49:29.420814 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:29.420712    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3afbe98f-5052-44e2-9d4b-da4bd2d30c3a-tmp-dir\") pod \"dns-default-d7m9q\" (UID: \"3afbe98f-5052-44e2-9d4b-da4bd2d30c3a\") " pod="openshift-dns/dns-default-d7m9q"
Sep 17 20:49:29.420961 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:29.420884    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9dda82c0-e203-412d-8d70-bffaf6285281-ca-trust-extracted\") pod \"image-registry-6c4b45cb74-s4gzt\" (UID: \"9dda82c0-e203-412d-8d70-bffaf6285281\") " pod="openshift-image-registry/image-registry-6c4b45cb74-s4gzt"
Sep 17 20:49:29.420961 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:29.420884    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9dda82c0-e203-412d-8d70-bffaf6285281-registry-certificates\") pod \"image-registry-6c4b45cb74-s4gzt\" (UID: \"9dda82c0-e203-412d-8d70-bffaf6285281\") " pod="openshift-image-registry/image-registry-6c4b45cb74-s4gzt"
Sep 17 20:49:29.420961 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:29.420889    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 20:49:29.420961 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:29.420944    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c4b45cb74-s4gzt: secret "image-registry-tls" not found
Sep 17 20:49:29.421134 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:29.421054    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9dda82c0-e203-412d-8d70-bffaf6285281-registry-tls podName:9dda82c0-e203-412d-8d70-bffaf6285281 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:29.921033967 +0000 UTC m=+35.073478331 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9dda82c0-e203-412d-8d70-bffaf6285281-registry-tls") pod "image-registry-6c4b45cb74-s4gzt" (UID: "9dda82c0-e203-412d-8d70-bffaf6285281") : secret "image-registry-tls" not found
Sep 17 20:49:29.421134 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:29.421108    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9dda82c0-e203-412d-8d70-bffaf6285281-trusted-ca\") pod \"image-registry-6c4b45cb74-s4gzt\" (UID: \"9dda82c0-e203-412d-8d70-bffaf6285281\") " pod="openshift-image-registry/image-registry-6c4b45cb74-s4gzt"
Sep 17 20:49:29.424568 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:29.424548    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9dda82c0-e203-412d-8d70-bffaf6285281-image-registry-private-configuration\") pod \"image-registry-6c4b45cb74-s4gzt\" (UID: \"9dda82c0-e203-412d-8d70-bffaf6285281\") " pod="openshift-image-registry/image-registry-6c4b45cb74-s4gzt"
Sep 17 20:49:29.424691 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:29.424551    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9dda82c0-e203-412d-8d70-bffaf6285281-installation-pull-secrets\") pod \"image-registry-6c4b45cb74-s4gzt\" (UID: \"9dda82c0-e203-412d-8d70-bffaf6285281\") " pod="openshift-image-registry/image-registry-6c4b45cb74-s4gzt"
Sep 17 20:49:29.429679 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:29.429657    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm9qs\" (UniqueName: \"kubernetes.io/projected/9dda82c0-e203-412d-8d70-bffaf6285281-kube-api-access-bm9qs\") pod \"image-registry-6c4b45cb74-s4gzt\" (UID: \"9dda82c0-e203-412d-8d70-bffaf6285281\") " pod="openshift-image-registry/image-registry-6c4b45cb74-s4gzt"
Sep 17 20:49:29.429900 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:29.429877    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9dda82c0-e203-412d-8d70-bffaf6285281-bound-sa-token\") pod \"image-registry-6c4b45cb74-s4gzt\" (UID: \"9dda82c0-e203-412d-8d70-bffaf6285281\") " pod="openshift-image-registry/image-registry-6c4b45cb74-s4gzt"
Sep 17 20:49:29.521625 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:29.521536    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kn79\" (UniqueName: \"kubernetes.io/projected/3afbe98f-5052-44e2-9d4b-da4bd2d30c3a-kube-api-access-4kn79\") pod \"dns-default-d7m9q\" (UID: \"3afbe98f-5052-44e2-9d4b-da4bd2d30c3a\") " pod="openshift-dns/dns-default-d7m9q"
Sep 17 20:49:29.521625 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:29.521579    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54ba4bcf-1779-4a9d-a5c3-d3bc4507466c-cert\") pod \"ingress-canary-z4lgm\" (UID: \"54ba4bcf-1779-4a9d-a5c3-d3bc4507466c\") " pod="openshift-ingress-canary/ingress-canary-z4lgm"
Sep 17 20:49:29.521625 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:29.521606    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwgbm\" (UniqueName: \"kubernetes.io/projected/54ba4bcf-1779-4a9d-a5c3-d3bc4507466c-kube-api-access-lwgbm\") pod \"ingress-canary-z4lgm\" (UID: \"54ba4bcf-1779-4a9d-a5c3-d3bc4507466c\") " pod="openshift-ingress-canary/ingress-canary-z4lgm"
Sep 17 20:49:29.521898 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:29.521640    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3afbe98f-5052-44e2-9d4b-da4bd2d30c3a-config-volume\") pod \"dns-default-d7m9q\" (UID: \"3afbe98f-5052-44e2-9d4b-da4bd2d30c3a\") " pod="openshift-dns/dns-default-d7m9q"
Sep 17 20:49:29.521898 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:29.521841    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3afbe98f-5052-44e2-9d4b-da4bd2d30c3a-metrics-tls\") pod \"dns-default-d7m9q\" (UID: \"3afbe98f-5052-44e2-9d4b-da4bd2d30c3a\") " pod="openshift-dns/dns-default-d7m9q"
Sep 17 20:49:29.521898 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:29.521888    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3afbe98f-5052-44e2-9d4b-da4bd2d30c3a-tmp-dir\") pod \"dns-default-d7m9q\" (UID: \"3afbe98f-5052-44e2-9d4b-da4bd2d30c3a\") " pod="openshift-dns/dns-default-d7m9q"
Sep 17 20:49:29.522066 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:29.522015    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 20:49:29.522120 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:29.522083    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3afbe98f-5052-44e2-9d4b-da4bd2d30c3a-metrics-tls podName:3afbe98f-5052-44e2-9d4b-da4bd2d30c3a nodeName:}" failed. No retries permitted until 2026-09-17 20:49:30.022067939 +0000 UTC m=+35.174512297 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3afbe98f-5052-44e2-9d4b-da4bd2d30c3a-metrics-tls") pod "dns-default-d7m9q" (UID: "3afbe98f-5052-44e2-9d4b-da4bd2d30c3a") : secret "dns-default-metrics-tls" not found
Sep 17 20:49:29.522192 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:29.522170    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3afbe98f-5052-44e2-9d4b-da4bd2d30c3a-tmp-dir\") pod \"dns-default-d7m9q\" (UID: \"3afbe98f-5052-44e2-9d4b-da4bd2d30c3a\") " pod="openshift-dns/dns-default-d7m9q"
Sep 17 20:49:29.522246 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:29.522233    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3afbe98f-5052-44e2-9d4b-da4bd2d30c3a-config-volume\") pod \"dns-default-d7m9q\" (UID: \"3afbe98f-5052-44e2-9d4b-da4bd2d30c3a\") " pod="openshift-dns/dns-default-d7m9q"
Sep 17 20:49:29.524257 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:29.524235    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54ba4bcf-1779-4a9d-a5c3-d3bc4507466c-cert\") pod \"ingress-canary-z4lgm\" (UID: \"54ba4bcf-1779-4a9d-a5c3-d3bc4507466c\") " pod="openshift-ingress-canary/ingress-canary-z4lgm"
Sep 17 20:49:29.531957 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:29.531934    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwgbm\" (UniqueName: \"kubernetes.io/projected/54ba4bcf-1779-4a9d-a5c3-d3bc4507466c-kube-api-access-lwgbm\") pod \"ingress-canary-z4lgm\" (UID: \"54ba4bcf-1779-4a9d-a5c3-d3bc4507466c\") " pod="openshift-ingress-canary/ingress-canary-z4lgm"
Sep 17 20:49:29.533512 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:29.533491    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kn79\" (UniqueName: \"kubernetes.io/projected/3afbe98f-5052-44e2-9d4b-da4bd2d30c3a-kube-api-access-4kn79\") pod \"dns-default-d7m9q\" (UID: \"3afbe98f-5052-44e2-9d4b-da4bd2d30c3a\") " pod="openshift-dns/dns-default-d7m9q"
Sep 17 20:49:29.552987 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:29.552947    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:49:29.571881 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:29.571842    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-z4lgm"
Sep 17 20:49:29.925245 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:29.925165    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9dda82c0-e203-412d-8d70-bffaf6285281-registry-tls\") pod \"image-registry-6c4b45cb74-s4gzt\" (UID: \"9dda82c0-e203-412d-8d70-bffaf6285281\") " pod="openshift-image-registry/image-registry-6c4b45cb74-s4gzt"
Sep 17 20:49:29.925386 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:29.925303    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 20:49:29.925386 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:29.925322    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c4b45cb74-s4gzt: secret "image-registry-tls" not found
Sep 17 20:49:29.925467 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:29.925391    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9dda82c0-e203-412d-8d70-bffaf6285281-registry-tls podName:9dda82c0-e203-412d-8d70-bffaf6285281 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:30.925362309 +0000 UTC m=+36.077806665 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9dda82c0-e203-412d-8d70-bffaf6285281-registry-tls") pod "image-registry-6c4b45cb74-s4gzt" (UID: "9dda82c0-e203-412d-8d70-bffaf6285281") : secret "image-registry-tls" not found
Sep 17 20:49:30.025937 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:30.025906    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3afbe98f-5052-44e2-9d4b-da4bd2d30c3a-metrics-tls\") pod \"dns-default-d7m9q\" (UID: \"3afbe98f-5052-44e2-9d4b-da4bd2d30c3a\") " pod="openshift-dns/dns-default-d7m9q"
Sep 17 20:49:30.026491 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:30.026083    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 20:49:30.026491 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:30.026170    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3afbe98f-5052-44e2-9d4b-da4bd2d30c3a-metrics-tls podName:3afbe98f-5052-44e2-9d4b-da4bd2d30c3a nodeName:}" failed. No retries permitted until 2026-09-17 20:49:31.026150269 +0000 UTC m=+36.178594628 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3afbe98f-5052-44e2-9d4b-da4bd2d30c3a-metrics-tls") pod "dns-default-d7m9q" (UID: "3afbe98f-5052-44e2-9d4b-da4bd2d30c3a") : secret "dns-default-metrics-tls" not found
Sep 17 20:49:30.351289 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:30.351255    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8b6dt"
Sep 17 20:49:30.351484 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:30.351378    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dpwhq"
Sep 17 20:49:30.351484 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:30.351254    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l28wp"
Sep 17 20:49:30.353756 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:30.353726    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8j64n\""
Sep 17 20:49:30.354817 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:30.354597    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vnpsw\""
Sep 17 20:49:30.354817 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:30.354620    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 20:49:30.354817 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:30.354665    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 20:49:30.354817 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:30.354665    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 20:49:30.354817 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:30.354713    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 20:49:30.933301 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:30.933270    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9dda82c0-e203-412d-8d70-bffaf6285281-registry-tls\") pod \"image-registry-6c4b45cb74-s4gzt\" (UID: \"9dda82c0-e203-412d-8d70-bffaf6285281\") " pod="openshift-image-registry/image-registry-6c4b45cb74-s4gzt"
Sep 17 20:49:30.933498 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:30.933472    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 20:49:30.933544 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:30.933504    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c4b45cb74-s4gzt: secret "image-registry-tls" not found
Sep 17 20:49:30.933584 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:30.933575    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9dda82c0-e203-412d-8d70-bffaf6285281-registry-tls podName:9dda82c0-e203-412d-8d70-bffaf6285281 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:32.933555577 +0000 UTC m=+38.085999948 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9dda82c0-e203-412d-8d70-bffaf6285281-registry-tls") pod "image-registry-6c4b45cb74-s4gzt" (UID: "9dda82c0-e203-412d-8d70-bffaf6285281") : secret "image-registry-tls" not found
Sep 17 20:49:31.034045 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:31.034004    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3afbe98f-5052-44e2-9d4b-da4bd2d30c3a-metrics-tls\") pod \"dns-default-d7m9q\" (UID: \"3afbe98f-5052-44e2-9d4b-da4bd2d30c3a\") " pod="openshift-dns/dns-default-d7m9q"
Sep 17 20:49:31.034475 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:31.034161    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 20:49:31.034475 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:31.034230    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3afbe98f-5052-44e2-9d4b-da4bd2d30c3a-metrics-tls podName:3afbe98f-5052-44e2-9d4b-da4bd2d30c3a nodeName:}" failed. No retries permitted until 2026-09-17 20:49:33.0342137 +0000 UTC m=+38.186658057 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3afbe98f-5052-44e2-9d4b-da4bd2d30c3a-metrics-tls") pod "dns-default-d7m9q" (UID: "3afbe98f-5052-44e2-9d4b-da4bd2d30c3a") : secret "dns-default-metrics-tls" not found
Sep 17 20:49:31.158020 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:31.157997    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-z4lgm"]
Sep 17 20:49:31.161831 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:49:31.161805    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54ba4bcf_1779_4a9d_a5c3_d3bc4507466c.slice/crio-c04c4416fd58b10532793f9bf716a0bf78f0f3a44978bd8c3ac3130e08ed96ac WatchSource:0}: Error finding container c04c4416fd58b10532793f9bf716a0bf78f0f3a44978bd8c3ac3130e08ed96ac: Status 404 returned error can't find the container with id c04c4416fd58b10532793f9bf716a0bf78f0f3a44978bd8c3ac3130e08ed96ac
Sep 17 20:49:31.543442 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:31.543258    2575 generic.go:356] "Generic (PLEG): container finished" podID="5ddbfff5-cd86-477e-9152-7b0969bb0054" containerID="0d882ac5770c9d2822e58b249acf128f94f28f9ae1b739b9bb83200cddeff844" exitCode=0
Sep 17 20:49:31.543613 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:31.543318    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zcsbp" event={"ID":"5ddbfff5-cd86-477e-9152-7b0969bb0054","Type":"ContainerDied","Data":"0d882ac5770c9d2822e58b249acf128f94f28f9ae1b739b9bb83200cddeff844"}
Sep 17 20:49:31.544645 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:31.544626    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-z4lgm" event={"ID":"54ba4bcf-1779-4a9d-a5c3-d3bc4507466c","Type":"ContainerStarted","Data":"c04c4416fd58b10532793f9bf716a0bf78f0f3a44978bd8c3ac3130e08ed96ac"}
Sep 17 20:49:32.551285 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:32.551255    2575 generic.go:356] "Generic (PLEG): container finished" podID="5ddbfff5-cd86-477e-9152-7b0969bb0054" containerID="e9b8186bc56c8d5bc33dc7ea469739b9c600644c69c06738b3f0ff9992164a33" exitCode=0
Sep 17 20:49:32.551723 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:32.551314    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zcsbp" event={"ID":"5ddbfff5-cd86-477e-9152-7b0969bb0054","Type":"ContainerDied","Data":"e9b8186bc56c8d5bc33dc7ea469739b9c600644c69c06738b3f0ff9992164a33"}
Sep 17 20:49:32.948485 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:32.948456    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9dda82c0-e203-412d-8d70-bffaf6285281-registry-tls\") pod \"image-registry-6c4b45cb74-s4gzt\" (UID: \"9dda82c0-e203-412d-8d70-bffaf6285281\") " pod="openshift-image-registry/image-registry-6c4b45cb74-s4gzt"
Sep 17 20:49:32.948644 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:32.948608    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 20:49:32.948644 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:32.948625    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c4b45cb74-s4gzt: secret "image-registry-tls" not found
Sep 17 20:49:32.948720 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:32.948678    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9dda82c0-e203-412d-8d70-bffaf6285281-registry-tls podName:9dda82c0-e203-412d-8d70-bffaf6285281 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:36.948664309 +0000 UTC m=+42.101108663 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9dda82c0-e203-412d-8d70-bffaf6285281-registry-tls") pod "image-registry-6c4b45cb74-s4gzt" (UID: "9dda82c0-e203-412d-8d70-bffaf6285281") : secret "image-registry-tls" not found
Sep 17 20:49:32.959252 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:32.959228    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:49:33.018493 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:33.018458    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:49:33.049373 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:33.049345    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3afbe98f-5052-44e2-9d4b-da4bd2d30c3a-metrics-tls\") pod \"dns-default-d7m9q\" (UID: \"3afbe98f-5052-44e2-9d4b-da4bd2d30c3a\") " pod="openshift-dns/dns-default-d7m9q"
Sep 17 20:49:33.049503 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:33.049488    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 20:49:33.049557 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:33.049546    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3afbe98f-5052-44e2-9d4b-da4bd2d30c3a-metrics-tls podName:3afbe98f-5052-44e2-9d4b-da4bd2d30c3a nodeName:}" failed. No retries permitted until 2026-09-17 20:49:37.049532999 +0000 UTC m=+42.201977358 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3afbe98f-5052-44e2-9d4b-da4bd2d30c3a-metrics-tls") pod "dns-default-d7m9q" (UID: "3afbe98f-5052-44e2-9d4b-da4bd2d30c3a") : secret "dns-default-metrics-tls" not found
Sep 17 20:49:33.558417 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:33.558371    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zcsbp" event={"ID":"5ddbfff5-cd86-477e-9152-7b0969bb0054","Type":"ContainerStarted","Data":"e8b90907266b37adc6ca5c17afabb21cb3301f90e95ff743fce260591a7fcb9b"}
Sep 17 20:49:33.585462 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:33.585203    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-zcsbp" podStartSLOduration=7.770410513 podStartE2EDuration="37.585183327s" totalImagesPullingTime="29.814772814s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:48:56 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:33.58494283 +0000 UTC m=+38.737387211" watchObservedRunningTime="2026-09-17 20:49:33.585183327 +0000 UTC m=+38.737627705"
Sep 17 20:49:33.728285 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:33.728246    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:49:33.823766 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:33.823708    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:49:33.924788 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:33.924758    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:49:33.989279 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:33.989236    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:49:34.307595 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:34.307565    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:49:34.409400 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:34.409369    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:49:34.562442 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:34.562358    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-z4lgm" event={"ID":"54ba4bcf-1779-4a9d-a5c3-d3bc4507466c","Type":"ContainerStarted","Data":"79a0c2074b87c82bf46b7b7ddc6db0c679a121a65c56c4c3c3f0456afd342b42"}
Sep 17 20:49:34.581466 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:34.581405    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-z4lgm" podStartSLOduration=2.346594981 podStartE2EDuration="5.581384009s" totalImagesPullingTime="3.234789028s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:49:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:34.579832583 +0000 UTC m=+39.732276961" watchObservedRunningTime="2026-09-17 20:49:34.581384009 +0000 UTC m=+39.733828386"
Sep 17 20:49:34.883287 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:34.883073    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:49:36.978394 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:36.978357    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9dda82c0-e203-412d-8d70-bffaf6285281-registry-tls\") pod \"image-registry-6c4b45cb74-s4gzt\" (UID: \"9dda82c0-e203-412d-8d70-bffaf6285281\") " pod="openshift-image-registry/image-registry-6c4b45cb74-s4gzt"
Sep 17 20:49:36.978825 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:36.978548    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 20:49:36.978825 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:36.978572    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c4b45cb74-s4gzt: secret "image-registry-tls" not found
Sep 17 20:49:36.978825 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:36.978659    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9dda82c0-e203-412d-8d70-bffaf6285281-registry-tls podName:9dda82c0-e203-412d-8d70-bffaf6285281 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:44.978636654 +0000 UTC m=+50.131081030 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9dda82c0-e203-412d-8d70-bffaf6285281-registry-tls") pod "image-registry-6c4b45cb74-s4gzt" (UID: "9dda82c0-e203-412d-8d70-bffaf6285281") : secret "image-registry-tls" not found
Sep 17 20:49:37.079186 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:37.079156    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3afbe98f-5052-44e2-9d4b-da4bd2d30c3a-metrics-tls\") pod \"dns-default-d7m9q\" (UID: \"3afbe98f-5052-44e2-9d4b-da4bd2d30c3a\") " pod="openshift-dns/dns-default-d7m9q"
Sep 17 20:49:37.079358 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:37.079289    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 20:49:37.079358 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:37.079349    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3afbe98f-5052-44e2-9d4b-da4bd2d30c3a-metrics-tls podName:3afbe98f-5052-44e2-9d4b-da4bd2d30c3a nodeName:}" failed. No retries permitted until 2026-09-17 20:49:45.079332788 +0000 UTC m=+50.231777156 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3afbe98f-5052-44e2-9d4b-da4bd2d30c3a-metrics-tls") pod "dns-default-d7m9q" (UID: "3afbe98f-5052-44e2-9d4b-da4bd2d30c3a") : secret "dns-default-metrics-tls" not found
Sep 17 20:49:40.549509 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:40.549477    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:49:40.649519 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:40.649471    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:49:43.065701 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:43.065661    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:49:43.156360 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:43.156331    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:49:43.220680 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:43.220642    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:49:44.104322 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:44.104292    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:49:44.174391 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:44.174353    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:49:45.032077 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:45.032044    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9dda82c0-e203-412d-8d70-bffaf6285281-registry-tls\") pod \"image-registry-6c4b45cb74-s4gzt\" (UID: \"9dda82c0-e203-412d-8d70-bffaf6285281\") " pod="openshift-image-registry/image-registry-6c4b45cb74-s4gzt"
Sep 17 20:49:45.032247 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:45.032189    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 20:49:45.032247 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:45.032206    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c4b45cb74-s4gzt: secret "image-registry-tls" not found
Sep 17 20:49:45.032316 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:45.032270    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9dda82c0-e203-412d-8d70-bffaf6285281-registry-tls podName:9dda82c0-e203-412d-8d70-bffaf6285281 nodeName:}" failed. No retries permitted until 2026-09-17 20:50:01.032255931 +0000 UTC m=+66.184700315 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9dda82c0-e203-412d-8d70-bffaf6285281-registry-tls") pod "image-registry-6c4b45cb74-s4gzt" (UID: "9dda82c0-e203-412d-8d70-bffaf6285281") : secret "image-registry-tls" not found
Sep 17 20:49:45.133258 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:45.133228    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3afbe98f-5052-44e2-9d4b-da4bd2d30c3a-metrics-tls\") pod \"dns-default-d7m9q\" (UID: \"3afbe98f-5052-44e2-9d4b-da4bd2d30c3a\") " pod="openshift-dns/dns-default-d7m9q"
Sep 17 20:49:45.133717 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:45.133400    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 20:49:45.133717 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:49:45.133493    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3afbe98f-5052-44e2-9d4b-da4bd2d30c3a-metrics-tls podName:3afbe98f-5052-44e2-9d4b-da4bd2d30c3a nodeName:}" failed. No retries permitted until 2026-09-17 20:50:01.133472779 +0000 UTC m=+66.285917181 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3afbe98f-5052-44e2-9d4b-da4bd2d30c3a-metrics-tls") pod "dns-default-d7m9q" (UID: "3afbe98f-5052-44e2-9d4b-da4bd2d30c3a") : secret "dns-default-metrics-tls" not found
Sep 17 20:49:45.158224 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:45.158193    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:49:45.258373 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:45.258346    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:49:47.122721 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:47.122682    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:49:49.286251 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:49.286211    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:49:49.351390 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:49.351351    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:49:50.006519 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:50.006488    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:49:50.092314 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:50.092287    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:49:55.537986 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:55.537955    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tmg6m"
Sep 17 20:49:55.703509 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:55.703480    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:49:55.810032 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:55.809957    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:49:56.308194 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:56.308162    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:49:56.405079 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:56.405050    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:49:59.285896 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:59.285862    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:49:59.380865 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:59.380820    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:49:59.485813 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:49:59.485780    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:50:01.045064 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:50:01.045022    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9dda82c0-e203-412d-8d70-bffaf6285281-registry-tls\") pod \"image-registry-6c4b45cb74-s4gzt\" (UID: \"9dda82c0-e203-412d-8d70-bffaf6285281\") " pod="openshift-image-registry/image-registry-6c4b45cb74-s4gzt"
Sep 17 20:50:01.045554 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:50:01.045126    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d42a790e-18b7-4478-b0c7-3c7cc4372fff-metrics-certs\") pod \"network-metrics-daemon-8b6dt\" (UID: \"d42a790e-18b7-4478-b0c7-3c7cc4372fff\") " pod="openshift-multus/network-metrics-daemon-8b6dt"
Sep 17 20:50:01.045554 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:50:01.045166    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 20:50:01.045554 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:50:01.045186    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c4b45cb74-s4gzt: secret "image-registry-tls" not found
Sep 17 20:50:01.045554 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:50:01.045248    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9dda82c0-e203-412d-8d70-bffaf6285281-registry-tls podName:9dda82c0-e203-412d-8d70-bffaf6285281 nodeName:}" failed. No retries permitted until 2026-09-17 20:50:33.045233503 +0000 UTC m=+98.197677857 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9dda82c0-e203-412d-8d70-bffaf6285281-registry-tls") pod "image-registry-6c4b45cb74-s4gzt" (UID: "9dda82c0-e203-412d-8d70-bffaf6285281") : secret "image-registry-tls" not found
Sep 17 20:50:01.047558 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:50:01.047537    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 20:50:01.056020 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:50:01.056000    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 20:50:01.056119 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:50:01.056061    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d42a790e-18b7-4478-b0c7-3c7cc4372fff-metrics-certs podName:d42a790e-18b7-4478-b0c7-3c7cc4372fff nodeName:}" failed. No retries permitted until 2026-09-17 20:51:05.056043153 +0000 UTC m=+130.208487519 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d42a790e-18b7-4478-b0c7-3c7cc4372fff-metrics-certs") pod "network-metrics-daemon-8b6dt" (UID: "d42a790e-18b7-4478-b0c7-3c7cc4372fff") : secret "metrics-daemon-secret" not found
Sep 17 20:50:01.146103 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:50:01.146062    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqcxv\" (UniqueName: \"kubernetes.io/projected/53dee621-a806-480a-80c6-ff174deb8ed4-kube-api-access-fqcxv\") pod \"network-check-target-l28wp\" (UID: \"53dee621-a806-480a-80c6-ff174deb8ed4\") " pod="openshift-network-diagnostics/network-check-target-l28wp"
Sep 17 20:50:01.146311 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:50:01.146121    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/334a13ac-ccb2-43eb-bd43-566981c058ff-original-pull-secret\") pod \"global-pull-secret-syncer-dpwhq\" (UID: \"334a13ac-ccb2-43eb-bd43-566981c058ff\") " pod="kube-system/global-pull-secret-syncer-dpwhq"
Sep 17 20:50:01.146311 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:50:01.146166    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3afbe98f-5052-44e2-9d4b-da4bd2d30c3a-metrics-tls\") pod \"dns-default-d7m9q\" (UID: \"3afbe98f-5052-44e2-9d4b-da4bd2d30c3a\") " pod="openshift-dns/dns-default-d7m9q"
Sep 17 20:50:01.146311 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:50:01.146259    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 20:50:01.146311 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:50:01.146305    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3afbe98f-5052-44e2-9d4b-da4bd2d30c3a-metrics-tls podName:3afbe98f-5052-44e2-9d4b-da4bd2d30c3a nodeName:}" failed. No retries permitted until 2026-09-17 20:50:33.146292743 +0000 UTC m=+98.298737097 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3afbe98f-5052-44e2-9d4b-da4bd2d30c3a-metrics-tls") pod "dns-default-d7m9q" (UID: "3afbe98f-5052-44e2-9d4b-da4bd2d30c3a") : secret "dns-default-metrics-tls" not found
Sep 17 20:50:01.149086 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:50:01.149058    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 20:50:01.149280 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:50:01.149229    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 20:50:01.158646 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:50:01.158621    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 20:50:01.159842 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:50:01.159822    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/334a13ac-ccb2-43eb-bd43-566981c058ff-original-pull-secret\") pod \"global-pull-secret-syncer-dpwhq\" (UID: \"334a13ac-ccb2-43eb-bd43-566981c058ff\") " pod="kube-system/global-pull-secret-syncer-dpwhq"
Sep 17 20:50:01.168882 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:50:01.168864    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqcxv\" (UniqueName: \"kubernetes.io/projected/53dee621-a806-480a-80c6-ff174deb8ed4-kube-api-access-fqcxv\") pod \"network-check-target-l28wp\" (UID: \"53dee621-a806-480a-80c6-ff174deb8ed4\") " pod="openshift-network-diagnostics/network-check-target-l28wp"
Sep 17 20:50:01.272892 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:50:01.272844    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dpwhq"
Sep 17 20:50:01.280707 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:50:01.280677    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vnpsw\""
Sep 17 20:50:01.288788 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:50:01.288767    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l28wp"
Sep 17 20:50:01.572680 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:50:01.572611    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dpwhq"]
Sep 17 20:50:01.576083 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:50:01.576058    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod334a13ac_ccb2_43eb_bd43_566981c058ff.slice/crio-4042c64a1327cc0e12ec9aff1060bde73bce817be7f2819d2297b21ea040a972 WatchSource:0}: Error finding container 4042c64a1327cc0e12ec9aff1060bde73bce817be7f2819d2297b21ea040a972: Status 404 returned error can't find the container with id 4042c64a1327cc0e12ec9aff1060bde73bce817be7f2819d2297b21ea040a972
Sep 17 20:50:01.592994 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:50:01.592962    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-l28wp"]
Sep 17 20:50:01.596496 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:50:01.596468    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53dee621_a806_480a_80c6_ff174deb8ed4.slice/crio-5ed51f8727c094dbc3db999cafb753d1fa02bf6e58b0b0ee62f6845022d5af2a WatchSource:0}: Error finding container 5ed51f8727c094dbc3db999cafb753d1fa02bf6e58b0b0ee62f6845022d5af2a: Status 404 returned error can't find the container with id 5ed51f8727c094dbc3db999cafb753d1fa02bf6e58b0b0ee62f6845022d5af2a
Sep 17 20:50:01.625567 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:50:01.625532    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-l28wp" event={"ID":"53dee621-a806-480a-80c6-ff174deb8ed4","Type":"ContainerStarted","Data":"5ed51f8727c094dbc3db999cafb753d1fa02bf6e58b0b0ee62f6845022d5af2a"}
Sep 17 20:50:01.626688 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:50:01.626635    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dpwhq" event={"ID":"334a13ac-ccb2-43eb-bd43-566981c058ff","Type":"ContainerStarted","Data":"4042c64a1327cc0e12ec9aff1060bde73bce817be7f2819d2297b21ea040a972"}
Sep 17 20:50:06.641292 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:50:06.641250    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-l28wp" event={"ID":"53dee621-a806-480a-80c6-ff174deb8ed4","Type":"ContainerStarted","Data":"3ed62f46528474334035897c14f571b18a6e861240ed3df5e399e7c7de3cb850"}
Sep 17 20:50:06.641733 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:50:06.641360    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-l28wp"
Sep 17 20:50:06.642872 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:50:06.642832    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dpwhq" event={"ID":"334a13ac-ccb2-43eb-bd43-566981c058ff","Type":"ContainerStarted","Data":"a342affa55294ca37750cc6a4e1eefb3570a398abfefa1e1daed2ed722e68f39"}
Sep 17 20:50:06.658160 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:50:06.658108    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-l28wp" podStartSLOduration=65.760042604 podStartE2EDuration="1m10.658091377s" totalImagesPullingTime="4.898048773s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:48:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:50:06.657623282 +0000 UTC m=+71.810067658" watchObservedRunningTime="2026-09-17 20:50:06.658091377 +0000 UTC m=+71.810535756"
Sep 17 20:50:10.975137 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:50:10.975106    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:50:11.081413 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:50:11.081374    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:50:21.167014 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:50:21.166871    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:50:21.254883 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:50:21.254828    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:50:31.622883 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:50:31.622815    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:50:31.723676 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:50:31.723641    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:50:33.074763 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:50:33.074723    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9dda82c0-e203-412d-8d70-bffaf6285281-registry-tls\") pod \"image-registry-6c4b45cb74-s4gzt\" (UID: \"9dda82c0-e203-412d-8d70-bffaf6285281\") " pod="openshift-image-registry/image-registry-6c4b45cb74-s4gzt"
Sep 17 20:50:33.075247 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:50:33.074914    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 20:50:33.075247 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:50:33.074935    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c4b45cb74-s4gzt: secret "image-registry-tls" not found
Sep 17 20:50:33.075247 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:50:33.075011    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9dda82c0-e203-412d-8d70-bffaf6285281-registry-tls podName:9dda82c0-e203-412d-8d70-bffaf6285281 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:37.074994086 +0000 UTC m=+162.227438439 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9dda82c0-e203-412d-8d70-bffaf6285281-registry-tls") pod "image-registry-6c4b45cb74-s4gzt" (UID: "9dda82c0-e203-412d-8d70-bffaf6285281") : secret "image-registry-tls" not found
Sep 17 20:50:33.176091 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:50:33.176035    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3afbe98f-5052-44e2-9d4b-da4bd2d30c3a-metrics-tls\") pod \"dns-default-d7m9q\" (UID: \"3afbe98f-5052-44e2-9d4b-da4bd2d30c3a\") " pod="openshift-dns/dns-default-d7m9q"
Sep 17 20:50:33.176262 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:50:33.176192    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 20:50:33.176303 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:50:33.176268    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3afbe98f-5052-44e2-9d4b-da4bd2d30c3a-metrics-tls podName:3afbe98f-5052-44e2-9d4b-da4bd2d30c3a nodeName:}" failed. No retries permitted until 2026-09-17 20:51:37.176253247 +0000 UTC m=+162.328697604 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3afbe98f-5052-44e2-9d4b-da4bd2d30c3a-metrics-tls") pod "dns-default-d7m9q" (UID: "3afbe98f-5052-44e2-9d4b-da4bd2d30c3a") : secret "dns-default-metrics-tls" not found
Sep 17 20:50:37.647611 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:50:37.647574    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-l28wp"
Sep 17 20:50:37.682811 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:50:37.682764    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-dpwhq" podStartSLOduration=96.764727849 podStartE2EDuration="1m41.682748317s" totalImagesPullingTime="4.918020468s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:48:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:50:06.676387125 +0000 UTC m=+71.828831491" watchObservedRunningTime="2026-09-17 20:50:37.682748317 +0000 UTC m=+102.835192692"
Sep 17 20:51:02.779139 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:02.779096    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:05.093059 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:05.092998    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d42a790e-18b7-4478-b0c7-3c7cc4372fff-metrics-certs\") pod \"network-metrics-daemon-8b6dt\" (UID: \"d42a790e-18b7-4478-b0c7-3c7cc4372fff\") " pod="openshift-multus/network-metrics-daemon-8b6dt"
Sep 17 20:51:05.093513 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:51:05.093132    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 20:51:05.093513 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:51:05.093187    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d42a790e-18b7-4478-b0c7-3c7cc4372fff-metrics-certs podName:d42a790e-18b7-4478-b0c7-3c7cc4372fff nodeName:}" failed. No retries permitted until 2026-09-17 20:53:07.093174104 +0000 UTC m=+252.245618458 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d42a790e-18b7-4478-b0c7-3c7cc4372fff-metrics-certs") pod "network-metrics-daemon-8b6dt" (UID: "d42a790e-18b7-4478-b0c7-3c7cc4372fff") : secret "metrics-daemon-secret" not found
Sep 17 20:51:05.439995 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:05.439914    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-jknl5"]
Sep 17 20:51:05.442586 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:05.442570    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-jknl5"
Sep 17 20:51:05.445573 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:05.445354    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 17 20:51:05.445573 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:05.445463    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 17 20:51:05.445573 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:05.445495    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 20:51:05.446610 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:05.446487    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-g5z64\""
Sep 17 20:51:05.446746 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:05.446659    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 20:51:05.450601 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:05.450581    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 17 20:51:05.455101 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:05.455078    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-8b75c8b68-8wx4k"]
Sep 17 20:51:05.457779 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:05.457762    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-jknl5"]
Sep 17 20:51:05.457908 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:05.457875    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-8b75c8b68-8wx4k"
Sep 17 20:51:05.460307 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:05.460284    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 17 20:51:05.460658 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:05.460640    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 17 20:51:05.460745 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:05.460724    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-tmxnn\""
Sep 17 20:51:05.460798 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:05.460749    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 17 20:51:05.460798 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:05.460784    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 17 20:51:05.460921 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:05.460751    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 17 20:51:05.461043 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:05.461029    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 17 20:51:05.477248 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:05.477229    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-8b75c8b68-8wx4k"]
Sep 17 20:51:05.496860 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:05.496826    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7e30e07a-26a2-4051-9dba-6170acb64eba-tmp\") pod \"insights-operator-6f9c644dd9-jknl5\" (UID: \"7e30e07a-26a2-4051-9dba-6170acb64eba\") " pod="openshift-insights/insights-operator-6f9c644dd9-jknl5"
Sep 17 20:51:05.496994 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:05.496893    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7e30e07a-26a2-4051-9dba-6170acb64eba-serving-cert\") pod \"insights-operator-6f9c644dd9-jknl5\" (UID: \"7e30e07a-26a2-4051-9dba-6170acb64eba\") " pod="openshift-insights/insights-operator-6f9c644dd9-jknl5"
Sep 17 20:51:05.496994 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:05.496951    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e30e07a-26a2-4051-9dba-6170acb64eba-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-jknl5\" (UID: \"7e30e07a-26a2-4051-9dba-6170acb64eba\") " pod="openshift-insights/insights-operator-6f9c644dd9-jknl5"
Sep 17 20:51:05.497116 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:05.497027    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/7e30e07a-26a2-4051-9dba-6170acb64eba-snapshots\") pod \"insights-operator-6f9c644dd9-jknl5\" (UID: \"7e30e07a-26a2-4051-9dba-6170acb64eba\") " pod="openshift-insights/insights-operator-6f9c644dd9-jknl5"
Sep 17 20:51:05.497116 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:05.497065    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e30e07a-26a2-4051-9dba-6170acb64eba-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-jknl5\" (UID: \"7e30e07a-26a2-4051-9dba-6170acb64eba\") " pod="openshift-insights/insights-operator-6f9c644dd9-jknl5"
Sep 17 20:51:05.497116 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:05.497094    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl5fw\" (UniqueName: \"kubernetes.io/projected/7e30e07a-26a2-4051-9dba-6170acb64eba-kube-api-access-vl5fw\") pod \"insights-operator-6f9c644dd9-jknl5\" (UID: \"7e30e07a-26a2-4051-9dba-6170acb64eba\") " pod="openshift-insights/insights-operator-6f9c644dd9-jknl5"
Sep 17 20:51:05.543923 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:05.543888    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-lm54d"]
Sep 17 20:51:05.546581 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:05.546564    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-lm54d"
Sep 17 20:51:05.551683 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:05.551652    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 17 20:51:05.551683 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:05.551675    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 17 20:51:05.551886 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:05.551657    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 20:51:05.551886 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:05.551660    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 20:51:05.551886 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:05.551749    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-qmdfr\""
Sep 17 20:51:05.561345 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:05.561322    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-lm54d"]
Sep 17 20:51:05.597744 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:05.597709    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e30e07a-26a2-4051-9dba-6170acb64eba-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-jknl5\" (UID: \"7e30e07a-26a2-4051-9dba-6170acb64eba\") " pod="openshift-insights/insights-operator-6f9c644dd9-jknl5"
Sep 17 20:51:05.597914 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:05.597765    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7eb327ea-1cee-4481-8533-423762e782e0-default-certificate\") pod \"router-default-8b75c8b68-8wx4k\" (UID: \"7eb327ea-1cee-4481-8533-423762e782e0\") " pod="openshift-ingress/router-default-8b75c8b68-8wx4k"
Sep 17 20:51:05.597914 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:05.597818    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7eb327ea-1cee-4481-8533-423762e782e0-service-ca-bundle\") pod \"router-default-8b75c8b68-8wx4k\" (UID: \"7eb327ea-1cee-4481-8533-423762e782e0\") " pod="openshift-ingress/router-default-8b75c8b68-8wx4k"
Sep 17 20:51:05.597996 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:05.597941    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/7e30e07a-26a2-4051-9dba-6170acb64eba-snapshots\") pod \"insights-operator-6f9c644dd9-jknl5\" (UID: \"7e30e07a-26a2-4051-9dba-6170acb64eba\") " pod="openshift-insights/insights-operator-6f9c644dd9-jknl5"
Sep 17 20:51:05.597996 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:05.597963    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e30e07a-26a2-4051-9dba-6170acb64eba-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-jknl5\" (UID: \"7e30e07a-26a2-4051-9dba-6170acb64eba\") " pod="openshift-insights/insights-operator-6f9c644dd9-jknl5"
Sep 17 20:51:05.597996 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:05.597978    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl5fw\" (UniqueName: \"kubernetes.io/projected/7e30e07a-26a2-4051-9dba-6170acb64eba-kube-api-access-vl5fw\") pod \"insights-operator-6f9c644dd9-jknl5\" (UID: \"7e30e07a-26a2-4051-9dba-6170acb64eba\") " pod="openshift-insights/insights-operator-6f9c644dd9-jknl5"
Sep 17 20:51:05.598135 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:05.598058    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7e30e07a-26a2-4051-9dba-6170acb64eba-tmp\") pod \"insights-operator-6f9c644dd9-jknl5\" (UID: \"7e30e07a-26a2-4051-9dba-6170acb64eba\") " pod="openshift-insights/insights-operator-6f9c644dd9-jknl5"
Sep 17 20:51:05.598135 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:05.598110    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7eb327ea-1cee-4481-8533-423762e782e0-stats-auth\") pod \"router-default-8b75c8b68-8wx4k\" (UID: \"7eb327ea-1cee-4481-8533-423762e782e0\") " pod="openshift-ingress/router-default-8b75c8b68-8wx4k"
Sep 17 20:51:05.598237 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:05.598143    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7eb327ea-1cee-4481-8533-423762e782e0-metrics-certs\") pod \"router-default-8b75c8b68-8wx4k\" (UID: \"7eb327ea-1cee-4481-8533-423762e782e0\") " pod="openshift-ingress/router-default-8b75c8b68-8wx4k"
Sep 17 20:51:05.598237 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:05.598205    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9d6r\" (UniqueName: \"kubernetes.io/projected/7eb327ea-1cee-4481-8533-423762e782e0-kube-api-access-d9d6r\") pod \"router-default-8b75c8b68-8wx4k\" (UID: \"7eb327ea-1cee-4481-8533-423762e782e0\") " pod="openshift-ingress/router-default-8b75c8b68-8wx4k"
Sep 17 20:51:05.598328 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:05.598241    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7e30e07a-26a2-4051-9dba-6170acb64eba-serving-cert\") pod \"insights-operator-6f9c644dd9-jknl5\" (UID: \"7e30e07a-26a2-4051-9dba-6170acb64eba\") " pod="openshift-insights/insights-operator-6f9c644dd9-jknl5"
Sep 17 20:51:05.598513 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:05.598490    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e30e07a-26a2-4051-9dba-6170acb64eba-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-jknl5\" (UID: \"7e30e07a-26a2-4051-9dba-6170acb64eba\") " pod="openshift-insights/insights-operator-6f9c644dd9-jknl5"
Sep 17 20:51:05.598597 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:05.598581    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7e30e07a-26a2-4051-9dba-6170acb64eba-tmp\") pod \"insights-operator-6f9c644dd9-jknl5\" (UID: \"7e30e07a-26a2-4051-9dba-6170acb64eba\") " pod="openshift-insights/insights-operator-6f9c644dd9-jknl5"
Sep 17 20:51:05.598650 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:05.598590    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/7e30e07a-26a2-4051-9dba-6170acb64eba-snapshots\") pod \"insights-operator-6f9c644dd9-jknl5\" (UID: \"7e30e07a-26a2-4051-9dba-6170acb64eba\") " pod="openshift-insights/insights-operator-6f9c644dd9-jknl5"
Sep 17 20:51:05.598691 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:05.598657    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e30e07a-26a2-4051-9dba-6170acb64eba-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-jknl5\" (UID: \"7e30e07a-26a2-4051-9dba-6170acb64eba\") " pod="openshift-insights/insights-operator-6f9c644dd9-jknl5"
Sep 17 20:51:05.600485 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:05.600464    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7e30e07a-26a2-4051-9dba-6170acb64eba-serving-cert\") pod \"insights-operator-6f9c644dd9-jknl5\" (UID: \"7e30e07a-26a2-4051-9dba-6170acb64eba\") " pod="openshift-insights/insights-operator-6f9c644dd9-jknl5"
Sep 17 20:51:05.604882 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:05.604834    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:05.606342 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:05.606324    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl5fw\" (UniqueName: \"kubernetes.io/projected/7e30e07a-26a2-4051-9dba-6170acb64eba-kube-api-access-vl5fw\") pod \"insights-operator-6f9c644dd9-jknl5\" (UID: \"7e30e07a-26a2-4051-9dba-6170acb64eba\") " pod="openshift-insights/insights-operator-6f9c644dd9-jknl5"
Sep 17 20:51:05.698898 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:05.698834    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a95f9e39-2b33-434d-b6d0-807da6f8f4d2-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-lm54d\" (UID: \"a95f9e39-2b33-434d-b6d0-807da6f8f4d2\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-lm54d"
Sep 17 20:51:05.699080 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:05.698910    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9rj8\" (UniqueName: \"kubernetes.io/projected/a95f9e39-2b33-434d-b6d0-807da6f8f4d2-kube-api-access-f9rj8\") pod \"cluster-monitoring-operator-847b8bd8d7-lm54d\" (UID: \"a95f9e39-2b33-434d-b6d0-807da6f8f4d2\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-lm54d"
Sep 17 20:51:05.699080 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:05.698942    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7eb327ea-1cee-4481-8533-423762e782e0-stats-auth\") pod \"router-default-8b75c8b68-8wx4k\" (UID: \"7eb327ea-1cee-4481-8533-423762e782e0\") " pod="openshift-ingress/router-default-8b75c8b68-8wx4k"
Sep 17 20:51:05.699080 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:05.698959    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7eb327ea-1cee-4481-8533-423762e782e0-metrics-certs\") pod \"router-default-8b75c8b68-8wx4k\" (UID: \"7eb327ea-1cee-4481-8533-423762e782e0\") " pod="openshift-ingress/router-default-8b75c8b68-8wx4k"
Sep 17 20:51:05.699080 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:05.699025    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9d6r\" (UniqueName: \"kubernetes.io/projected/7eb327ea-1cee-4481-8533-423762e782e0-kube-api-access-d9d6r\") pod \"router-default-8b75c8b68-8wx4k\" (UID: \"7eb327ea-1cee-4481-8533-423762e782e0\") " pod="openshift-ingress/router-default-8b75c8b68-8wx4k"
Sep 17 20:51:05.699080 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:05.699063    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a95f9e39-2b33-434d-b6d0-807da6f8f4d2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-lm54d\" (UID: \"a95f9e39-2b33-434d-b6d0-807da6f8f4d2\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-lm54d"
Sep 17 20:51:05.699320 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:51:05.699086    2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 20:51:05.699320 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:05.699133    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7eb327ea-1cee-4481-8533-423762e782e0-default-certificate\") pod \"router-default-8b75c8b68-8wx4k\" (UID: \"7eb327ea-1cee-4481-8533-423762e782e0\") " pod="openshift-ingress/router-default-8b75c8b68-8wx4k"
Sep 17 20:51:05.699320 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:51:05.699148    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7eb327ea-1cee-4481-8533-423762e782e0-metrics-certs podName:7eb327ea-1cee-4481-8533-423762e782e0 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:06.199130907 +0000 UTC m=+131.351575288 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7eb327ea-1cee-4481-8533-423762e782e0-metrics-certs") pod "router-default-8b75c8b68-8wx4k" (UID: "7eb327ea-1cee-4481-8533-423762e782e0") : secret "router-metrics-certs-default" not found
Sep 17 20:51:05.699320 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:05.699174    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7eb327ea-1cee-4481-8533-423762e782e0-service-ca-bundle\") pod \"router-default-8b75c8b68-8wx4k\" (UID: \"7eb327ea-1cee-4481-8533-423762e782e0\") " pod="openshift-ingress/router-default-8b75c8b68-8wx4k"
Sep 17 20:51:05.699508 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:51:05.699321    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7eb327ea-1cee-4481-8533-423762e782e0-service-ca-bundle podName:7eb327ea-1cee-4481-8533-423762e782e0 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:06.199304901 +0000 UTC m=+131.351749275 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7eb327ea-1cee-4481-8533-423762e782e0-service-ca-bundle") pod "router-default-8b75c8b68-8wx4k" (UID: "7eb327ea-1cee-4481-8533-423762e782e0") : configmap references non-existent config key: service-ca.crt
Sep 17 20:51:05.701464 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:05.701438    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7eb327ea-1cee-4481-8533-423762e782e0-stats-auth\") pod \"router-default-8b75c8b68-8wx4k\" (UID: \"7eb327ea-1cee-4481-8533-423762e782e0\") " pod="openshift-ingress/router-default-8b75c8b68-8wx4k"
Sep 17 20:51:05.701601 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:05.701578    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7eb327ea-1cee-4481-8533-423762e782e0-default-certificate\") pod \"router-default-8b75c8b68-8wx4k\" (UID: \"7eb327ea-1cee-4481-8533-423762e782e0\") " pod="openshift-ingress/router-default-8b75c8b68-8wx4k"
Sep 17 20:51:05.706504 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:05.706473    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:05.708049 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:05.708021    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9d6r\" (UniqueName: \"kubernetes.io/projected/7eb327ea-1cee-4481-8533-423762e782e0-kube-api-access-d9d6r\") pod \"router-default-8b75c8b68-8wx4k\" (UID: \"7eb327ea-1cee-4481-8533-423762e782e0\") " pod="openshift-ingress/router-default-8b75c8b68-8wx4k"
Sep 17 20:51:05.754801 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:05.754770    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-jknl5"
Sep 17 20:51:05.800082 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:05.800050    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a95f9e39-2b33-434d-b6d0-807da6f8f4d2-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-lm54d\" (UID: \"a95f9e39-2b33-434d-b6d0-807da6f8f4d2\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-lm54d"
Sep 17 20:51:05.800246 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:05.800090    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9rj8\" (UniqueName: \"kubernetes.io/projected/a95f9e39-2b33-434d-b6d0-807da6f8f4d2-kube-api-access-f9rj8\") pod \"cluster-monitoring-operator-847b8bd8d7-lm54d\" (UID: \"a95f9e39-2b33-434d-b6d0-807da6f8f4d2\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-lm54d"
Sep 17 20:51:05.800246 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:05.800150    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a95f9e39-2b33-434d-b6d0-807da6f8f4d2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-lm54d\" (UID: \"a95f9e39-2b33-434d-b6d0-807da6f8f4d2\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-lm54d"
Sep 17 20:51:05.800361 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:51:05.800343    2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 20:51:05.800422 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:51:05.800410    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a95f9e39-2b33-434d-b6d0-807da6f8f4d2-cluster-monitoring-operator-tls podName:a95f9e39-2b33-434d-b6d0-807da6f8f4d2 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:06.300388641 +0000 UTC m=+131.452832995 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a95f9e39-2b33-434d-b6d0-807da6f8f4d2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-lm54d" (UID: "a95f9e39-2b33-434d-b6d0-807da6f8f4d2") : secret "cluster-monitoring-operator-tls" not found
Sep 17 20:51:05.801518 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:05.801268    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a95f9e39-2b33-434d-b6d0-807da6f8f4d2-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-lm54d\" (UID: \"a95f9e39-2b33-434d-b6d0-807da6f8f4d2\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-lm54d"
Sep 17 20:51:05.808715 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:05.808680    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9rj8\" (UniqueName: \"kubernetes.io/projected/a95f9e39-2b33-434d-b6d0-807da6f8f4d2-kube-api-access-f9rj8\") pod \"cluster-monitoring-operator-847b8bd8d7-lm54d\" (UID: \"a95f9e39-2b33-434d-b6d0-807da6f8f4d2\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-lm54d"
Sep 17 20:51:06.031921 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:06.031831    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-jknl5"]
Sep 17 20:51:06.035050 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:51:06.035027    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e30e07a_26a2_4051_9dba_6170acb64eba.slice/crio-541b2b73129b248f2d0e41c582ce00156aaaf21ea340bcf63571f263595922b1 WatchSource:0}: Error finding container 541b2b73129b248f2d0e41c582ce00156aaaf21ea340bcf63571f263595922b1: Status 404 returned error can't find the container with id 541b2b73129b248f2d0e41c582ce00156aaaf21ea340bcf63571f263595922b1
Sep 17 20:51:06.202717 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:06.202684    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7eb327ea-1cee-4481-8533-423762e782e0-metrics-certs\") pod \"router-default-8b75c8b68-8wx4k\" (UID: \"7eb327ea-1cee-4481-8533-423762e782e0\") " pod="openshift-ingress/router-default-8b75c8b68-8wx4k"
Sep 17 20:51:06.203132 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:06.202834    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7eb327ea-1cee-4481-8533-423762e782e0-service-ca-bundle\") pod \"router-default-8b75c8b68-8wx4k\" (UID: \"7eb327ea-1cee-4481-8533-423762e782e0\") " pod="openshift-ingress/router-default-8b75c8b68-8wx4k"
Sep 17 20:51:06.203132 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:51:06.202840    2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 20:51:06.203132 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:51:06.202922    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7eb327ea-1cee-4481-8533-423762e782e0-metrics-certs podName:7eb327ea-1cee-4481-8533-423762e782e0 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:07.202905354 +0000 UTC m=+132.355349727 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7eb327ea-1cee-4481-8533-423762e782e0-metrics-certs") pod "router-default-8b75c8b68-8wx4k" (UID: "7eb327ea-1cee-4481-8533-423762e782e0") : secret "router-metrics-certs-default" not found
Sep 17 20:51:06.203132 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:51:06.202984    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7eb327ea-1cee-4481-8533-423762e782e0-service-ca-bundle podName:7eb327ea-1cee-4481-8533-423762e782e0 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:07.202973127 +0000 UTC m=+132.355417481 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7eb327ea-1cee-4481-8533-423762e782e0-service-ca-bundle") pod "router-default-8b75c8b68-8wx4k" (UID: "7eb327ea-1cee-4481-8533-423762e782e0") : configmap references non-existent config key: service-ca.crt
Sep 17 20:51:06.303678 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:06.303588    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a95f9e39-2b33-434d-b6d0-807da6f8f4d2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-lm54d\" (UID: \"a95f9e39-2b33-434d-b6d0-807da6f8f4d2\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-lm54d"
Sep 17 20:51:06.303820 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:51:06.303737    2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 20:51:06.303820 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:51:06.303803    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a95f9e39-2b33-434d-b6d0-807da6f8f4d2-cluster-monitoring-operator-tls podName:a95f9e39-2b33-434d-b6d0-807da6f8f4d2 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:07.303786676 +0000 UTC m=+132.456231029 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a95f9e39-2b33-434d-b6d0-807da6f8f4d2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-lm54d" (UID: "a95f9e39-2b33-434d-b6d0-807da6f8f4d2") : secret "cluster-monitoring-operator-tls" not found
Sep 17 20:51:06.789307 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:06.789260    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-jknl5" event={"ID":"7e30e07a-26a2-4051-9dba-6170acb64eba","Type":"ContainerStarted","Data":"541b2b73129b248f2d0e41c582ce00156aaaf21ea340bcf63571f263595922b1"}
Sep 17 20:51:07.210076 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:07.210039    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7eb327ea-1cee-4481-8533-423762e782e0-metrics-certs\") pod \"router-default-8b75c8b68-8wx4k\" (UID: \"7eb327ea-1cee-4481-8533-423762e782e0\") " pod="openshift-ingress/router-default-8b75c8b68-8wx4k"
Sep 17 20:51:07.210489 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:07.210240    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7eb327ea-1cee-4481-8533-423762e782e0-service-ca-bundle\") pod \"router-default-8b75c8b68-8wx4k\" (UID: \"7eb327ea-1cee-4481-8533-423762e782e0\") " pod="openshift-ingress/router-default-8b75c8b68-8wx4k"
Sep 17 20:51:07.210489 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:51:07.210239    2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 20:51:07.210489 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:51:07.210343    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7eb327ea-1cee-4481-8533-423762e782e0-service-ca-bundle podName:7eb327ea-1cee-4481-8533-423762e782e0 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:09.210322619 +0000 UTC m=+134.362766992 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7eb327ea-1cee-4481-8533-423762e782e0-service-ca-bundle") pod "router-default-8b75c8b68-8wx4k" (UID: "7eb327ea-1cee-4481-8533-423762e782e0") : configmap references non-existent config key: service-ca.crt
Sep 17 20:51:07.210489 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:51:07.210394    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7eb327ea-1cee-4481-8533-423762e782e0-metrics-certs podName:7eb327ea-1cee-4481-8533-423762e782e0 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:09.210373112 +0000 UTC m=+134.362817467 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7eb327ea-1cee-4481-8533-423762e782e0-metrics-certs") pod "router-default-8b75c8b68-8wx4k" (UID: "7eb327ea-1cee-4481-8533-423762e782e0") : secret "router-metrics-certs-default" not found
Sep 17 20:51:07.311690 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:07.311648    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a95f9e39-2b33-434d-b6d0-807da6f8f4d2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-lm54d\" (UID: \"a95f9e39-2b33-434d-b6d0-807da6f8f4d2\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-lm54d"
Sep 17 20:51:07.311870 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:51:07.311784    2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 20:51:07.311954 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:51:07.311887    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a95f9e39-2b33-434d-b6d0-807da6f8f4d2-cluster-monitoring-operator-tls podName:a95f9e39-2b33-434d-b6d0-807da6f8f4d2 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:09.31186293 +0000 UTC m=+134.464307301 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a95f9e39-2b33-434d-b6d0-807da6f8f4d2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-lm54d" (UID: "a95f9e39-2b33-434d-b6d0-807da6f8f4d2") : secret "cluster-monitoring-operator-tls" not found
Sep 17 20:51:07.793627 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:07.793588    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-jknl5" event={"ID":"7e30e07a-26a2-4051-9dba-6170acb64eba","Type":"ContainerStarted","Data":"5f18a47765127888a6fb3cec4c9bbca017b4234700777938cb1199756c277bad"}
Sep 17 20:51:07.810736 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:07.810687    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-6f9c644dd9-jknl5" podStartSLOduration=1.159222601 podStartE2EDuration="2.81066945s" totalImagesPullingTime="1.651446849s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:51:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:51:07.810404641 +0000 UTC m=+132.962849021" watchObservedRunningTime="2026-09-17 20:51:07.81066945 +0000 UTC m=+132.963113827"
Sep 17 20:51:09.091233 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:09.091196    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-qsltg"]
Sep 17 20:51:09.094517 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:09.094493    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qsltg"
Sep 17 20:51:09.096641 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:09.096621    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 20:51:09.096888 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:09.096865    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 20:51:09.097473 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:09.097453    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-p9nvv\""
Sep 17 20:51:09.107180 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:09.107158    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qsltg"]
Sep 17 20:51:09.111523 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:09.111495    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:09.175964 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:09.175922    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:09.226214 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:09.226177    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7eb327ea-1cee-4481-8533-423762e782e0-metrics-certs\") pod \"router-default-8b75c8b68-8wx4k\" (UID: \"7eb327ea-1cee-4481-8533-423762e782e0\") " pod="openshift-ingress/router-default-8b75c8b68-8wx4k"
Sep 17 20:51:09.226389 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:09.226221    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fc616903-063e-4c1c-b98e-472c4be8203f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qsltg\" (UID: \"fc616903-063e-4c1c-b98e-472c4be8203f\") " pod="openshift-insights/insights-runtime-extractor-qsltg"
Sep 17 20:51:09.226389 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:09.226241    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fc616903-063e-4c1c-b98e-472c4be8203f-crio-socket\") pod \"insights-runtime-extractor-qsltg\" (UID: \"fc616903-063e-4c1c-b98e-472c4be8203f\") " pod="openshift-insights/insights-runtime-extractor-qsltg"
Sep 17 20:51:09.226389 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:09.226298    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7eb327ea-1cee-4481-8533-423762e782e0-service-ca-bundle\") pod \"router-default-8b75c8b68-8wx4k\" (UID: \"7eb327ea-1cee-4481-8533-423762e782e0\") " pod="openshift-ingress/router-default-8b75c8b68-8wx4k"
Sep 17 20:51:09.226389 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:09.226319    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fc616903-063e-4c1c-b98e-472c4be8203f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qsltg\" (UID: \"fc616903-063e-4c1c-b98e-472c4be8203f\") " pod="openshift-insights/insights-runtime-extractor-qsltg"
Sep 17 20:51:09.226389 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:09.226365    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq4xz\" (UniqueName: \"kubernetes.io/projected/fc616903-063e-4c1c-b98e-472c4be8203f-kube-api-access-hq4xz\") pod \"insights-runtime-extractor-qsltg\" (UID: \"fc616903-063e-4c1c-b98e-472c4be8203f\") " pod="openshift-insights/insights-runtime-extractor-qsltg"
Sep 17 20:51:09.226389 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:51:09.226376    2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 20:51:09.226682 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:09.226404    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fc616903-063e-4c1c-b98e-472c4be8203f-data-volume\") pod \"insights-runtime-extractor-qsltg\" (UID: \"fc616903-063e-4c1c-b98e-472c4be8203f\") " pod="openshift-insights/insights-runtime-extractor-qsltg"
Sep 17 20:51:09.226682 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:51:09.226445    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7eb327ea-1cee-4481-8533-423762e782e0-metrics-certs podName:7eb327ea-1cee-4481-8533-423762e782e0 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:13.226424231 +0000 UTC m=+138.378868601 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7eb327ea-1cee-4481-8533-423762e782e0-metrics-certs") pod "router-default-8b75c8b68-8wx4k" (UID: "7eb327ea-1cee-4481-8533-423762e782e0") : secret "router-metrics-certs-default" not found
Sep 17 20:51:09.226682 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:51:09.226468    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7eb327ea-1cee-4481-8533-423762e782e0-service-ca-bundle podName:7eb327ea-1cee-4481-8533-423762e782e0 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:13.226456379 +0000 UTC m=+138.378900737 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7eb327ea-1cee-4481-8533-423762e782e0-service-ca-bundle") pod "router-default-8b75c8b68-8wx4k" (UID: "7eb327ea-1cee-4481-8533-423762e782e0") : configmap references non-existent config key: service-ca.crt
Sep 17 20:51:09.327199 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:09.327157    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fc616903-063e-4c1c-b98e-472c4be8203f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qsltg\" (UID: \"fc616903-063e-4c1c-b98e-472c4be8203f\") " pod="openshift-insights/insights-runtime-extractor-qsltg"
Sep 17 20:51:09.327199 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:09.327200    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fc616903-063e-4c1c-b98e-472c4be8203f-crio-socket\") pod \"insights-runtime-extractor-qsltg\" (UID: \"fc616903-063e-4c1c-b98e-472c4be8203f\") " pod="openshift-insights/insights-runtime-extractor-qsltg"
Sep 17 20:51:09.327450 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:09.327241    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fc616903-063e-4c1c-b98e-472c4be8203f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qsltg\" (UID: \"fc616903-063e-4c1c-b98e-472c4be8203f\") " pod="openshift-insights/insights-runtime-extractor-qsltg"
Sep 17 20:51:09.327450 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:09.327303    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq4xz\" (UniqueName: \"kubernetes.io/projected/fc616903-063e-4c1c-b98e-472c4be8203f-kube-api-access-hq4xz\") pod \"insights-runtime-extractor-qsltg\" (UID: \"fc616903-063e-4c1c-b98e-472c4be8203f\") " pod="openshift-insights/insights-runtime-extractor-qsltg"
Sep 17 20:51:09.327450 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:51:09.327320    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 20:51:09.327450 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:09.327335    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a95f9e39-2b33-434d-b6d0-807da6f8f4d2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-lm54d\" (UID: \"a95f9e39-2b33-434d-b6d0-807da6f8f4d2\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-lm54d"
Sep 17 20:51:09.327450 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:51:09.327366    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 20:51:09.327450 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:51:09.327384    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc616903-063e-4c1c-b98e-472c4be8203f-insights-runtime-extractor-tls podName:fc616903-063e-4c1c-b98e-472c4be8203f nodeName:}" failed. No retries permitted until 2026-09-17 20:51:09.827366181 +0000 UTC m=+134.979810554 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fc616903-063e-4c1c-b98e-472c4be8203f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qsltg" (UID: "fc616903-063e-4c1c-b98e-472c4be8203f") : secret "insights-runtime-extractor-tls" not found
Sep 17 20:51:09.327450 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:09.327406    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fc616903-063e-4c1c-b98e-472c4be8203f-data-volume\") pod \"insights-runtime-extractor-qsltg\" (UID: \"fc616903-063e-4c1c-b98e-472c4be8203f\") " pod="openshift-insights/insights-runtime-extractor-qsltg"
Sep 17 20:51:09.327450 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:51:09.327433    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fc616903-063e-4c1c-b98e-472c4be8203f-kube-rbac-proxy-cm podName:fc616903-063e-4c1c-b98e-472c4be8203f nodeName:}" failed. No retries permitted until 2026-09-17 20:51:09.827415575 +0000 UTC m=+134.979859932 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fc616903-063e-4c1c-b98e-472c4be8203f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qsltg" (UID: "fc616903-063e-4c1c-b98e-472c4be8203f") : configmap "kube-rbac-proxy" not found
Sep 17 20:51:09.327450 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:51:09.327442    2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 20:51:09.327450 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:09.327370    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fc616903-063e-4c1c-b98e-472c4be8203f-crio-socket\") pod \"insights-runtime-extractor-qsltg\" (UID: \"fc616903-063e-4c1c-b98e-472c4be8203f\") " pod="openshift-insights/insights-runtime-extractor-qsltg"
Sep 17 20:51:09.327769 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:51:09.327483    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a95f9e39-2b33-434d-b6d0-807da6f8f4d2-cluster-monitoring-operator-tls podName:a95f9e39-2b33-434d-b6d0-807da6f8f4d2 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:13.327472772 +0000 UTC m=+138.479917138 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a95f9e39-2b33-434d-b6d0-807da6f8f4d2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-lm54d" (UID: "a95f9e39-2b33-434d-b6d0-807da6f8f4d2") : secret "cluster-monitoring-operator-tls" not found
Sep 17 20:51:09.327769 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:09.327693    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fc616903-063e-4c1c-b98e-472c4be8203f-data-volume\") pod \"insights-runtime-extractor-qsltg\" (UID: \"fc616903-063e-4c1c-b98e-472c4be8203f\") " pod="openshift-insights/insights-runtime-extractor-qsltg"
Sep 17 20:51:09.338945 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:09.338924    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq4xz\" (UniqueName: \"kubernetes.io/projected/fc616903-063e-4c1c-b98e-472c4be8203f-kube-api-access-hq4xz\") pod \"insights-runtime-extractor-qsltg\" (UID: \"fc616903-063e-4c1c-b98e-472c4be8203f\") " pod="openshift-insights/insights-runtime-extractor-qsltg"
Sep 17 20:51:09.833265 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:09.833231    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fc616903-063e-4c1c-b98e-472c4be8203f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qsltg\" (UID: \"fc616903-063e-4c1c-b98e-472c4be8203f\") " pod="openshift-insights/insights-runtime-extractor-qsltg"
Sep 17 20:51:09.833265 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:09.833281    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fc616903-063e-4c1c-b98e-472c4be8203f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qsltg\" (UID: \"fc616903-063e-4c1c-b98e-472c4be8203f\") " pod="openshift-insights/insights-runtime-extractor-qsltg"
Sep 17 20:51:09.833502 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:51:09.833356    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 20:51:09.833502 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:51:09.833385    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 20:51:09.833502 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:51:09.833411    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fc616903-063e-4c1c-b98e-472c4be8203f-kube-rbac-proxy-cm podName:fc616903-063e-4c1c-b98e-472c4be8203f nodeName:}" failed. No retries permitted until 2026-09-17 20:51:10.833395619 +0000 UTC m=+135.985839972 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fc616903-063e-4c1c-b98e-472c4be8203f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qsltg" (UID: "fc616903-063e-4c1c-b98e-472c4be8203f") : configmap "kube-rbac-proxy" not found
Sep 17 20:51:09.833502 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:51:09.833440    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc616903-063e-4c1c-b98e-472c4be8203f-insights-runtime-extractor-tls podName:fc616903-063e-4c1c-b98e-472c4be8203f nodeName:}" failed. No retries permitted until 2026-09-17 20:51:10.833426898 +0000 UTC m=+135.985871257 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fc616903-063e-4c1c-b98e-472c4be8203f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qsltg" (UID: "fc616903-063e-4c1c-b98e-472c4be8203f") : secret "insights-runtime-extractor-tls" not found
Sep 17 20:51:10.839010 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:10.838980    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-48p9f_5af35da6-3bf6-49ca-9a78-0218c7166c83/dns-node-resolver/0.log"
Sep 17 20:51:10.842350 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:10.842321    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fc616903-063e-4c1c-b98e-472c4be8203f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qsltg\" (UID: \"fc616903-063e-4c1c-b98e-472c4be8203f\") " pod="openshift-insights/insights-runtime-extractor-qsltg"
Sep 17 20:51:10.842480 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:10.842373    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fc616903-063e-4c1c-b98e-472c4be8203f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qsltg\" (UID: \"fc616903-063e-4c1c-b98e-472c4be8203f\") " pod="openshift-insights/insights-runtime-extractor-qsltg"
Sep 17 20:51:10.842526 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:51:10.842479    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 20:51:10.842557 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:51:10.842535    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fc616903-063e-4c1c-b98e-472c4be8203f-kube-rbac-proxy-cm podName:fc616903-063e-4c1c-b98e-472c4be8203f nodeName:}" failed. No retries permitted until 2026-09-17 20:51:12.84252231 +0000 UTC m=+137.994966678 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fc616903-063e-4c1c-b98e-472c4be8203f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qsltg" (UID: "fc616903-063e-4c1c-b98e-472c4be8203f") : configmap "kube-rbac-proxy" not found
Sep 17 20:51:10.842595 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:51:10.842481    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 20:51:10.842642 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:51:10.842628    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc616903-063e-4c1c-b98e-472c4be8203f-insights-runtime-extractor-tls podName:fc616903-063e-4c1c-b98e-472c4be8203f nodeName:}" failed. No retries permitted until 2026-09-17 20:51:12.842616279 +0000 UTC m=+137.995060646 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fc616903-063e-4c1c-b98e-472c4be8203f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qsltg" (UID: "fc616903-063e-4c1c-b98e-472c4be8203f") : secret "insights-runtime-extractor-tls" not found
Sep 17 20:51:11.484793 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:11.484759    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-55dn9"]
Sep 17 20:51:11.487916 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:11.487899    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-55dn9"
Sep 17 20:51:11.490880 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:11.490840    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 17 20:51:11.491211 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:11.491191    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 17 20:51:11.492055 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:11.492034    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 17 20:51:11.492146 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:11.492078    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-pkcsf\""
Sep 17 20:51:11.499295 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:11.499275    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-55dn9"]
Sep 17 20:51:11.638353 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:11.638325    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2lrgx_0369efa0-b31a-4fa5-b426-95ba067ea731/node-ca/0.log"
Sep 17 20:51:11.650115 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:11.650082    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9l9z\" (UniqueName: \"kubernetes.io/projected/0c11c2c9-0efb-4522-89c3-6682fb58daa6-kube-api-access-n9l9z\") pod \"cluster-samples-operator-685d7d68b5-55dn9\" (UID: \"0c11c2c9-0efb-4522-89c3-6682fb58daa6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-55dn9"
Sep 17 20:51:11.650249 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:11.650127    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0c11c2c9-0efb-4522-89c3-6682fb58daa6-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-55dn9\" (UID: \"0c11c2c9-0efb-4522-89c3-6682fb58daa6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-55dn9"
Sep 17 20:51:11.750795 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:11.750715    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9l9z\" (UniqueName: \"kubernetes.io/projected/0c11c2c9-0efb-4522-89c3-6682fb58daa6-kube-api-access-n9l9z\") pod \"cluster-samples-operator-685d7d68b5-55dn9\" (UID: \"0c11c2c9-0efb-4522-89c3-6682fb58daa6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-55dn9"
Sep 17 20:51:11.750795 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:11.750760    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0c11c2c9-0efb-4522-89c3-6682fb58daa6-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-55dn9\" (UID: \"0c11c2c9-0efb-4522-89c3-6682fb58daa6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-55dn9"
Sep 17 20:51:11.750986 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:51:11.750954    2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 20:51:11.751051 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:51:11.751038    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c11c2c9-0efb-4522-89c3-6682fb58daa6-samples-operator-tls podName:0c11c2c9-0efb-4522-89c3-6682fb58daa6 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:12.251020473 +0000 UTC m=+137.403464828 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0c11c2c9-0efb-4522-89c3-6682fb58daa6-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-55dn9" (UID: "0c11c2c9-0efb-4522-89c3-6682fb58daa6") : secret "samples-operator-tls" not found
Sep 17 20:51:11.761286 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:11.761258    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9l9z\" (UniqueName: \"kubernetes.io/projected/0c11c2c9-0efb-4522-89c3-6682fb58daa6-kube-api-access-n9l9z\") pod \"cluster-samples-operator-685d7d68b5-55dn9\" (UID: \"0c11c2c9-0efb-4522-89c3-6682fb58daa6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-55dn9"
Sep 17 20:51:12.255256 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:12.255224    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0c11c2c9-0efb-4522-89c3-6682fb58daa6-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-55dn9\" (UID: \"0c11c2c9-0efb-4522-89c3-6682fb58daa6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-55dn9"
Sep 17 20:51:12.255636 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:51:12.255374    2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 20:51:12.255636 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:51:12.255441    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c11c2c9-0efb-4522-89c3-6682fb58daa6-samples-operator-tls podName:0c11c2c9-0efb-4522-89c3-6682fb58daa6 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:13.255422606 +0000 UTC m=+138.407866961 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0c11c2c9-0efb-4522-89c3-6682fb58daa6-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-55dn9" (UID: "0c11c2c9-0efb-4522-89c3-6682fb58daa6") : secret "samples-operator-tls" not found
Sep 17 20:51:12.841281 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:12.841255    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-z4lgm_54ba4bcf-1779-4a9d-a5c3-d3bc4507466c/serve-healthcheck-canary/0.log"
Sep 17 20:51:12.860486 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:12.860455    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fc616903-063e-4c1c-b98e-472c4be8203f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qsltg\" (UID: \"fc616903-063e-4c1c-b98e-472c4be8203f\") " pod="openshift-insights/insights-runtime-extractor-qsltg"
Sep 17 20:51:12.860613 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:12.860500    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fc616903-063e-4c1c-b98e-472c4be8203f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qsltg\" (UID: \"fc616903-063e-4c1c-b98e-472c4be8203f\") " pod="openshift-insights/insights-runtime-extractor-qsltg"
Sep 17 20:51:12.860613 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:51:12.860573    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 20:51:12.860613 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:51:12.860598    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 20:51:12.860749 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:51:12.860631    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fc616903-063e-4c1c-b98e-472c4be8203f-kube-rbac-proxy-cm podName:fc616903-063e-4c1c-b98e-472c4be8203f nodeName:}" failed. No retries permitted until 2026-09-17 20:51:16.860618115 +0000 UTC m=+142.013062481 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fc616903-063e-4c1c-b98e-472c4be8203f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qsltg" (UID: "fc616903-063e-4c1c-b98e-472c4be8203f") : configmap "kube-rbac-proxy" not found
Sep 17 20:51:12.860749 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:51:12.860659    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc616903-063e-4c1c-b98e-472c4be8203f-insights-runtime-extractor-tls podName:fc616903-063e-4c1c-b98e-472c4be8203f nodeName:}" failed. No retries permitted until 2026-09-17 20:51:16.860645469 +0000 UTC m=+142.013089837 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fc616903-063e-4c1c-b98e-472c4be8203f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qsltg" (UID: "fc616903-063e-4c1c-b98e-472c4be8203f") : secret "insights-runtime-extractor-tls" not found
Sep 17 20:51:13.264303 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:13.264264    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7eb327ea-1cee-4481-8533-423762e782e0-service-ca-bundle\") pod \"router-default-8b75c8b68-8wx4k\" (UID: \"7eb327ea-1cee-4481-8533-423762e782e0\") " pod="openshift-ingress/router-default-8b75c8b68-8wx4k"
Sep 17 20:51:13.264688 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:13.264383    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0c11c2c9-0efb-4522-89c3-6682fb58daa6-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-55dn9\" (UID: \"0c11c2c9-0efb-4522-89c3-6682fb58daa6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-55dn9"
Sep 17 20:51:13.264688 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:13.264405    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7eb327ea-1cee-4481-8533-423762e782e0-metrics-certs\") pod \"router-default-8b75c8b68-8wx4k\" (UID: \"7eb327ea-1cee-4481-8533-423762e782e0\") " pod="openshift-ingress/router-default-8b75c8b68-8wx4k"
Sep 17 20:51:13.264688 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:51:13.264439    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7eb327ea-1cee-4481-8533-423762e782e0-service-ca-bundle podName:7eb327ea-1cee-4481-8533-423762e782e0 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:21.264421743 +0000 UTC m=+146.416866096 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7eb327ea-1cee-4481-8533-423762e782e0-service-ca-bundle") pod "router-default-8b75c8b68-8wx4k" (UID: "7eb327ea-1cee-4481-8533-423762e782e0") : configmap references non-existent config key: service-ca.crt
Sep 17 20:51:13.264688 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:51:13.264500    2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 20:51:13.264688 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:51:13.264530    2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 20:51:13.264688 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:51:13.264556    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7eb327ea-1cee-4481-8533-423762e782e0-metrics-certs podName:7eb327ea-1cee-4481-8533-423762e782e0 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:21.264540443 +0000 UTC m=+146.416984797 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7eb327ea-1cee-4481-8533-423762e782e0-metrics-certs") pod "router-default-8b75c8b68-8wx4k" (UID: "7eb327ea-1cee-4481-8533-423762e782e0") : secret "router-metrics-certs-default" not found
Sep 17 20:51:13.264688 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:51:13.264572    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c11c2c9-0efb-4522-89c3-6682fb58daa6-samples-operator-tls podName:0c11c2c9-0efb-4522-89c3-6682fb58daa6 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:15.26456188 +0000 UTC m=+140.417006238 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0c11c2c9-0efb-4522-89c3-6682fb58daa6-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-55dn9" (UID: "0c11c2c9-0efb-4522-89c3-6682fb58daa6") : secret "samples-operator-tls" not found
Sep 17 20:51:13.365302 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:13.365266    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a95f9e39-2b33-434d-b6d0-807da6f8f4d2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-lm54d\" (UID: \"a95f9e39-2b33-434d-b6d0-807da6f8f4d2\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-lm54d"
Sep 17 20:51:13.365445 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:51:13.365376    2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 20:51:13.365445 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:51:13.365435    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a95f9e39-2b33-434d-b6d0-807da6f8f4d2-cluster-monitoring-operator-tls podName:a95f9e39-2b33-434d-b6d0-807da6f8f4d2 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:21.365418724 +0000 UTC m=+146.517863080 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a95f9e39-2b33-434d-b6d0-807da6f8f4d2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-lm54d" (UID: "a95f9e39-2b33-434d-b6d0-807da6f8f4d2") : secret "cluster-monitoring-operator-tls" not found
Sep 17 20:51:14.212013 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:14.211976    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:14.290219 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:14.290179    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:15.279984 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:15.279931    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0c11c2c9-0efb-4522-89c3-6682fb58daa6-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-55dn9\" (UID: \"0c11c2c9-0efb-4522-89c3-6682fb58daa6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-55dn9"
Sep 17 20:51:15.280206 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:51:15.280109    2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 20:51:15.280206 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:51:15.280179    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c11c2c9-0efb-4522-89c3-6682fb58daa6-samples-operator-tls podName:0c11c2c9-0efb-4522-89c3-6682fb58daa6 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:19.280159089 +0000 UTC m=+144.432603458 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0c11c2c9-0efb-4522-89c3-6682fb58daa6-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-55dn9" (UID: "0c11c2c9-0efb-4522-89c3-6682fb58daa6") : secret "samples-operator-tls" not found
Sep 17 20:51:15.437410 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:15.437375    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-9mj28"]
Sep 17 20:51:15.440532 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:15.440515    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-9mj28"
Sep 17 20:51:15.443007 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:15.442969    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 17 20:51:15.443007 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:15.442987    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 17 20:51:15.443199 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:15.442988    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 17 20:51:15.443199 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:15.442979    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-8z8s2\""
Sep 17 20:51:15.443650 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:15.443636    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 17 20:51:15.450001 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:15.449979    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-9mj28"]
Sep 17 20:51:15.551979 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:15.551899    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-v4g7j"]
Sep 17 20:51:15.554900 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:15.554885    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-v4g7j"
Sep 17 20:51:15.557269 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:15.557239    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 17 20:51:15.557269 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:15.557249    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-5bx77\""
Sep 17 20:51:15.557470 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:15.557292    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 17 20:51:15.557470 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:15.557311    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 17 20:51:15.557565 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:15.557552    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 17 20:51:15.562589 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:15.562559    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-v4g7j"]
Sep 17 20:51:15.582983 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:15.582955    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/08f37200-5c00-4946-8f1f-4b640e804e32-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-9mj28\" (UID: \"08f37200-5c00-4946-8f1f-4b640e804e32\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-9mj28"
Sep 17 20:51:15.583118 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:15.583012    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/08f37200-5c00-4946-8f1f-4b640e804e32-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-9mj28\" (UID: \"08f37200-5c00-4946-8f1f-4b640e804e32\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-9mj28"
Sep 17 20:51:15.583118 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:15.583065    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vth2b\" (UniqueName: \"kubernetes.io/projected/08f37200-5c00-4946-8f1f-4b640e804e32-kube-api-access-vth2b\") pod \"kube-storage-version-migrator-operator-55889f7b47-9mj28\" (UID: \"08f37200-5c00-4946-8f1f-4b640e804e32\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-9mj28"
Sep 17 20:51:15.684003 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:15.683950    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bc9n\" (UniqueName: \"kubernetes.io/projected/2d3961cf-90ef-46f5-b7e4-7af64d4d183e-kube-api-access-7bc9n\") pod \"service-ca-operator-57cdc4b768-v4g7j\" (UID: \"2d3961cf-90ef-46f5-b7e4-7af64d4d183e\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-v4g7j"
Sep 17 20:51:15.684003 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:15.684005    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/08f37200-5c00-4946-8f1f-4b640e804e32-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-9mj28\" (UID: \"08f37200-5c00-4946-8f1f-4b640e804e32\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-9mj28"
Sep 17 20:51:15.684003 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:15.684027    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2d3961cf-90ef-46f5-b7e4-7af64d4d183e-serving-cert\") pod \"service-ca-operator-57cdc4b768-v4g7j\" (UID: \"2d3961cf-90ef-46f5-b7e4-7af64d4d183e\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-v4g7j"
Sep 17 20:51:15.684307 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:15.684046    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2d3961cf-90ef-46f5-b7e4-7af64d4d183e-config\") pod \"service-ca-operator-57cdc4b768-v4g7j\" (UID: \"2d3961cf-90ef-46f5-b7e4-7af64d4d183e\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-v4g7j"
Sep 17 20:51:15.684307 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:15.684092    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/08f37200-5c00-4946-8f1f-4b640e804e32-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-9mj28\" (UID: \"08f37200-5c00-4946-8f1f-4b640e804e32\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-9mj28"
Sep 17 20:51:15.684307 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:15.684108    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vth2b\" (UniqueName: \"kubernetes.io/projected/08f37200-5c00-4946-8f1f-4b640e804e32-kube-api-access-vth2b\") pod \"kube-storage-version-migrator-operator-55889f7b47-9mj28\" (UID: \"08f37200-5c00-4946-8f1f-4b640e804e32\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-9mj28"
Sep 17 20:51:15.684573 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:15.684553    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/08f37200-5c00-4946-8f1f-4b640e804e32-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-9mj28\" (UID: \"08f37200-5c00-4946-8f1f-4b640e804e32\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-9mj28"
Sep 17 20:51:15.686295 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:15.686275    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/08f37200-5c00-4946-8f1f-4b640e804e32-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-9mj28\" (UID: \"08f37200-5c00-4946-8f1f-4b640e804e32\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-9mj28"
Sep 17 20:51:15.692523 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:15.692498    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vth2b\" (UniqueName: \"kubernetes.io/projected/08f37200-5c00-4946-8f1f-4b640e804e32-kube-api-access-vth2b\") pod \"kube-storage-version-migrator-operator-55889f7b47-9mj28\" (UID: \"08f37200-5c00-4946-8f1f-4b640e804e32\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-9mj28"
Sep 17 20:51:15.749048 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:15.749011    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-9mj28"
Sep 17 20:51:15.785247 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:15.785206    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bc9n\" (UniqueName: \"kubernetes.io/projected/2d3961cf-90ef-46f5-b7e4-7af64d4d183e-kube-api-access-7bc9n\") pod \"service-ca-operator-57cdc4b768-v4g7j\" (UID: \"2d3961cf-90ef-46f5-b7e4-7af64d4d183e\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-v4g7j"
Sep 17 20:51:15.785432 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:15.785336    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2d3961cf-90ef-46f5-b7e4-7af64d4d183e-serving-cert\") pod \"service-ca-operator-57cdc4b768-v4g7j\" (UID: \"2d3961cf-90ef-46f5-b7e4-7af64d4d183e\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-v4g7j"
Sep 17 20:51:15.785432 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:15.785368    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2d3961cf-90ef-46f5-b7e4-7af64d4d183e-config\") pod \"service-ca-operator-57cdc4b768-v4g7j\" (UID: \"2d3961cf-90ef-46f5-b7e4-7af64d4d183e\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-v4g7j"
Sep 17 20:51:15.785980 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:15.785957    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2d3961cf-90ef-46f5-b7e4-7af64d4d183e-config\") pod \"service-ca-operator-57cdc4b768-v4g7j\" (UID: \"2d3961cf-90ef-46f5-b7e4-7af64d4d183e\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-v4g7j"
Sep 17 20:51:15.787984 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:15.787953    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2d3961cf-90ef-46f5-b7e4-7af64d4d183e-serving-cert\") pod \"service-ca-operator-57cdc4b768-v4g7j\" (UID: \"2d3961cf-90ef-46f5-b7e4-7af64d4d183e\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-v4g7j"
Sep 17 20:51:15.798347 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:15.798323    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bc9n\" (UniqueName: \"kubernetes.io/projected/2d3961cf-90ef-46f5-b7e4-7af64d4d183e-kube-api-access-7bc9n\") pod \"service-ca-operator-57cdc4b768-v4g7j\" (UID: \"2d3961cf-90ef-46f5-b7e4-7af64d4d183e\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-v4g7j"
Sep 17 20:51:15.864998 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:15.864904    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-v4g7j"
Sep 17 20:51:16.020951 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:16.020911    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-9mj28"]
Sep 17 20:51:16.023945 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:51:16.023918    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08f37200_5c00_4946_8f1f_4b640e804e32.slice/crio-8c6e7606679f91912a7b2641bcd58e81035f3c3124962b9d0687772864e57c11 WatchSource:0}: Error finding container 8c6e7606679f91912a7b2641bcd58e81035f3c3124962b9d0687772864e57c11: Status 404 returned error can't find the container with id 8c6e7606679f91912a7b2641bcd58e81035f3c3124962b9d0687772864e57c11
Sep 17 20:51:16.133165 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:16.133053    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-v4g7j"]
Sep 17 20:51:16.136729 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:51:16.136700    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d3961cf_90ef_46f5_b7e4_7af64d4d183e.slice/crio-e31f89c019329019f3eb664e5dcf107fb326cc1c7ccd90acd2cbb51f89d82c36 WatchSource:0}: Error finding container e31f89c019329019f3eb664e5dcf107fb326cc1c7ccd90acd2cbb51f89d82c36: Status 404 returned error can't find the container with id e31f89c019329019f3eb664e5dcf107fb326cc1c7ccd90acd2cbb51f89d82c36
Sep 17 20:51:16.816017 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:16.815976    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-v4g7j" event={"ID":"2d3961cf-90ef-46f5-b7e4-7af64d4d183e","Type":"ContainerStarted","Data":"e31f89c019329019f3eb664e5dcf107fb326cc1c7ccd90acd2cbb51f89d82c36"}
Sep 17 20:51:16.817348 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:16.817305    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-9mj28" event={"ID":"08f37200-5c00-4946-8f1f-4b640e804e32","Type":"ContainerStarted","Data":"8c6e7606679f91912a7b2641bcd58e81035f3c3124962b9d0687772864e57c11"}
Sep 17 20:51:16.894396 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:16.894356    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fc616903-063e-4c1c-b98e-472c4be8203f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qsltg\" (UID: \"fc616903-063e-4c1c-b98e-472c4be8203f\") " pod="openshift-insights/insights-runtime-extractor-qsltg"
Sep 17 20:51:16.894575 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:16.894427    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fc616903-063e-4c1c-b98e-472c4be8203f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qsltg\" (UID: \"fc616903-063e-4c1c-b98e-472c4be8203f\") " pod="openshift-insights/insights-runtime-extractor-qsltg"
Sep 17 20:51:16.894575 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:51:16.894520    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 20:51:16.894682 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:51:16.894605    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc616903-063e-4c1c-b98e-472c4be8203f-insights-runtime-extractor-tls podName:fc616903-063e-4c1c-b98e-472c4be8203f nodeName:}" failed. No retries permitted until 2026-09-17 20:51:24.894580587 +0000 UTC m=+150.047024958 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fc616903-063e-4c1c-b98e-472c4be8203f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qsltg" (UID: "fc616903-063e-4c1c-b98e-472c4be8203f") : secret "insights-runtime-extractor-tls" not found
Sep 17 20:51:16.894682 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:51:16.894526    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 20:51:16.894785 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:51:16.894690    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fc616903-063e-4c1c-b98e-472c4be8203f-kube-rbac-proxy-cm podName:fc616903-063e-4c1c-b98e-472c4be8203f nodeName:}" failed. No retries permitted until 2026-09-17 20:51:24.894667518 +0000 UTC m=+150.047111888 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fc616903-063e-4c1c-b98e-472c4be8203f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qsltg" (UID: "fc616903-063e-4c1c-b98e-472c4be8203f") : configmap "kube-rbac-proxy" not found
Sep 17 20:51:18.825595 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:18.825554    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-9mj28" event={"ID":"08f37200-5c00-4946-8f1f-4b640e804e32","Type":"ContainerStarted","Data":"f411581fdab280b7f995255423d4ca967764c902545900674b75b79737413adc"}
Sep 17 20:51:18.827089 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:18.827062    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-v4g7j" event={"ID":"2d3961cf-90ef-46f5-b7e4-7af64d4d183e","Type":"ContainerStarted","Data":"4b79b51d3aac0fce1c544efe69b91ed060e8219a39d53a603f58dc0ad0ba46a3"}
Sep 17 20:51:18.842487 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:18.842429    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-9mj28" podStartSLOduration=1.5135479109999999 podStartE2EDuration="3.842409458s" totalImagesPullingTime="2.328861547s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:51:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:51:18.842258065 +0000 UTC m=+143.994702444" watchObservedRunningTime="2026-09-17 20:51:18.842409458 +0000 UTC m=+143.994853834"
Sep 17 20:51:18.860071 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:18.860018    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-v4g7j" podStartSLOduration=1.637649867 podStartE2EDuration="3.859997117s" totalImagesPullingTime="2.22234725s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:51:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:51:18.858692072 +0000 UTC m=+144.011136449" watchObservedRunningTime="2026-09-17 20:51:18.859997117 +0000 UTC m=+144.012441494"
Sep 17 20:51:19.318453 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:19.318403    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0c11c2c9-0efb-4522-89c3-6682fb58daa6-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-55dn9\" (UID: \"0c11c2c9-0efb-4522-89c3-6682fb58daa6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-55dn9"
Sep 17 20:51:19.318627 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:51:19.318549    2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 20:51:19.318627 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:51:19.318616    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c11c2c9-0efb-4522-89c3-6682fb58daa6-samples-operator-tls podName:0c11c2c9-0efb-4522-89c3-6682fb58daa6 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:27.318599148 +0000 UTC m=+152.471043501 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0c11c2c9-0efb-4522-89c3-6682fb58daa6-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-55dn9" (UID: "0c11c2c9-0efb-4522-89c3-6682fb58daa6") : secret "samples-operator-tls" not found
Sep 17 20:51:19.859627 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:19.859587    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:19.977583 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:19.977546    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:20.077543 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:20.077511    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:20.285598 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:20.285562    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:20.354835 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:20.354804    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:20.577251 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:20.577164    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:20.668039 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:20.668005    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:21.339061 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:21.339020    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7eb327ea-1cee-4481-8533-423762e782e0-metrics-certs\") pod \"router-default-8b75c8b68-8wx4k\" (UID: \"7eb327ea-1cee-4481-8533-423762e782e0\") " pod="openshift-ingress/router-default-8b75c8b68-8wx4k"
Sep 17 20:51:21.339403 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:21.339074    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7eb327ea-1cee-4481-8533-423762e782e0-service-ca-bundle\") pod \"router-default-8b75c8b68-8wx4k\" (UID: \"7eb327ea-1cee-4481-8533-423762e782e0\") " pod="openshift-ingress/router-default-8b75c8b68-8wx4k"
Sep 17 20:51:21.339403 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:51:21.339182    2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 20:51:21.339403 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:51:21.339205    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7eb327ea-1cee-4481-8533-423762e782e0-service-ca-bundle podName:7eb327ea-1cee-4481-8533-423762e782e0 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:37.339189773 +0000 UTC m=+162.491634132 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7eb327ea-1cee-4481-8533-423762e782e0-service-ca-bundle") pod "router-default-8b75c8b68-8wx4k" (UID: "7eb327ea-1cee-4481-8533-423762e782e0") : configmap references non-existent config key: service-ca.crt
Sep 17 20:51:21.339403 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:51:21.339237    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7eb327ea-1cee-4481-8533-423762e782e0-metrics-certs podName:7eb327ea-1cee-4481-8533-423762e782e0 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:37.339224647 +0000 UTC m=+162.491669000 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7eb327ea-1cee-4481-8533-423762e782e0-metrics-certs") pod "router-default-8b75c8b68-8wx4k" (UID: "7eb327ea-1cee-4481-8533-423762e782e0") : secret "router-metrics-certs-default" not found
Sep 17 20:51:21.440154 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:21.440096    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a95f9e39-2b33-434d-b6d0-807da6f8f4d2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-lm54d\" (UID: \"a95f9e39-2b33-434d-b6d0-807da6f8f4d2\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-lm54d"
Sep 17 20:51:21.440336 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:51:21.440252    2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 20:51:21.440336 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:51:21.440320    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a95f9e39-2b33-434d-b6d0-807da6f8f4d2-cluster-monitoring-operator-tls podName:a95f9e39-2b33-434d-b6d0-807da6f8f4d2 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:37.440304653 +0000 UTC m=+162.592749007 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a95f9e39-2b33-434d-b6d0-807da6f8f4d2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-lm54d" (UID: "a95f9e39-2b33-434d-b6d0-807da6f8f4d2") : secret "cluster-monitoring-operator-tls" not found
Sep 17 20:51:22.230601 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:22.230575    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:22.391280 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:22.391244    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:22.473920 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:22.473888    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:24.017750 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:24.017717    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:24.972101 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:24.972056    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fc616903-063e-4c1c-b98e-472c4be8203f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qsltg\" (UID: \"fc616903-063e-4c1c-b98e-472c4be8203f\") " pod="openshift-insights/insights-runtime-extractor-qsltg"
Sep 17 20:51:24.972258 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:24.972203    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fc616903-063e-4c1c-b98e-472c4be8203f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qsltg\" (UID: \"fc616903-063e-4c1c-b98e-472c4be8203f\") " pod="openshift-insights/insights-runtime-extractor-qsltg"
Sep 17 20:51:24.972258 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:51:24.972209    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 20:51:24.972370 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:51:24.972278    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc616903-063e-4c1c-b98e-472c4be8203f-insights-runtime-extractor-tls podName:fc616903-063e-4c1c-b98e-472c4be8203f nodeName:}" failed. No retries permitted until 2026-09-17 20:51:40.972262649 +0000 UTC m=+166.124707008 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fc616903-063e-4c1c-b98e-472c4be8203f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qsltg" (UID: "fc616903-063e-4c1c-b98e-472c4be8203f") : secret "insights-runtime-extractor-tls" not found
Sep 17 20:51:24.972724 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:24.972704    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fc616903-063e-4c1c-b98e-472c4be8203f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qsltg\" (UID: \"fc616903-063e-4c1c-b98e-472c4be8203f\") " pod="openshift-insights/insights-runtime-extractor-qsltg"
Sep 17 20:51:25.510119 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:25.510077    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:25.605041 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:25.605002    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:25.703761 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:25.703729    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:25.805179 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:25.805105    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:25.903964 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:25.903931    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:26.211296 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:26.211264    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:26.814053 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:26.814024    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:27.225506 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:27.225475    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:27.390256 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:27.390227    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0c11c2c9-0efb-4522-89c3-6682fb58daa6-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-55dn9\" (UID: \"0c11c2c9-0efb-4522-89c3-6682fb58daa6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-55dn9"
Sep 17 20:51:27.392640 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:27.392619    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0c11c2c9-0efb-4522-89c3-6682fb58daa6-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-55dn9\" (UID: \"0c11c2c9-0efb-4522-89c3-6682fb58daa6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-55dn9"
Sep 17 20:51:27.397382 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:27.397361    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-55dn9"
Sep 17 20:51:27.681399 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:27.681329    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-55dn9"]
Sep 17 20:51:27.847508 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:27.847474    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:27.860799 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:27.860767    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-55dn9" event={"ID":"0c11c2c9-0efb-4522-89c3-6682fb58daa6","Type":"ContainerStarted","Data":"bce8c1c5cbe37e2567b8c63c2d3c5b81479cee42efef29570b26556a160f6070"}
Sep 17 20:51:27.998507 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:27.998477    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:28.206954 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:28.206868    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:28.594879 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:28.594831    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:28.796681 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:28.796636    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:29.214414 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:29.214367    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:29.403918 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:29.403892    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:29.791715 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:29.791680    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:29.868260 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:29.868230    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-55dn9" event={"ID":"0c11c2c9-0efb-4522-89c3-6682fb58daa6","Type":"ContainerStarted","Data":"7186dade13af28d1b188297c33eca41257205b290647c6033e63067e54a0361e"}
Sep 17 20:51:29.868431 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:29.868283    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-55dn9" event={"ID":"0c11c2c9-0efb-4522-89c3-6682fb58daa6","Type":"ContainerStarted","Data":"5626cce087c6fba11fa88d8d1a53f12096f6a0ea065ef3c48785f9f9ae9581f4"}
Sep 17 20:51:29.886769 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:29.886727    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-55dn9" podStartSLOduration=17.287745972 podStartE2EDuration="18.886713633s" totalImagesPullingTime="1.598967661s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:51:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:51:29.885191961 +0000 UTC m=+155.037636348" watchObservedRunningTime="2026-09-17 20:51:29.886713633 +0000 UTC m=+155.039158009"
Sep 17 20:51:29.892983 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:29.892946    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:30.383482 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:30.383449    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:30.589214 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:30.589184    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:30.677988 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:30.677906    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:30.794288 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:30.794257    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:30.984660 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:30.984629    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:31.992079 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:31.992040    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:32.196537 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:32.196509    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:32.251882 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:51:32.251803    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-6c4b45cb74-s4gzt" podUID="9dda82c0-e203-412d-8d70-bffaf6285281"
Sep 17 20:51:32.293008 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:51:32.292980    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-d7m9q" podUID="3afbe98f-5052-44e2-9d4b-da4bd2d30c3a"
Sep 17 20:51:32.605174 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:32.605101    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:32.811140 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:32.811109    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:32.978182 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:32.978148    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:33.365514 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:51:33.365440    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-8b6dt" podUID="d42a790e-18b7-4478-b0c7-3c7cc4372fff"
Sep 17 20:51:33.385400 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:33.385373    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:33.785361 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:33.785321    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:33.983127 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:33.983092    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:34.389985 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:34.389952    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:35.202994 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:35.202963    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:35.391142 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:35.391099    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:35.490664 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:35.490578    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:35.798631 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:35.798563    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:35.996329 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:35.996301    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:36.207487 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:36.207461    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:37.170758 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:37.170723    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9dda82c0-e203-412d-8d70-bffaf6285281-registry-tls\") pod \"image-registry-6c4b45cb74-s4gzt\" (UID: \"9dda82c0-e203-412d-8d70-bffaf6285281\") " pod="openshift-image-registry/image-registry-6c4b45cb74-s4gzt"
Sep 17 20:51:37.173080 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:37.173057    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9dda82c0-e203-412d-8d70-bffaf6285281-registry-tls\") pod \"image-registry-6c4b45cb74-s4gzt\" (UID: \"9dda82c0-e203-412d-8d70-bffaf6285281\") " pod="openshift-image-registry/image-registry-6c4b45cb74-s4gzt"
Sep 17 20:51:37.271558 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:37.271529    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3afbe98f-5052-44e2-9d4b-da4bd2d30c3a-metrics-tls\") pod \"dns-default-d7m9q\" (UID: \"3afbe98f-5052-44e2-9d4b-da4bd2d30c3a\") " pod="openshift-dns/dns-default-d7m9q"
Sep 17 20:51:37.273577 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:37.273553    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3afbe98f-5052-44e2-9d4b-da4bd2d30c3a-metrics-tls\") pod \"dns-default-d7m9q\" (UID: \"3afbe98f-5052-44e2-9d4b-da4bd2d30c3a\") " pod="openshift-dns/dns-default-d7m9q"
Sep 17 20:51:37.372769 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:37.372739    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7eb327ea-1cee-4481-8533-423762e782e0-service-ca-bundle\") pod \"router-default-8b75c8b68-8wx4k\" (UID: \"7eb327ea-1cee-4481-8533-423762e782e0\") " pod="openshift-ingress/router-default-8b75c8b68-8wx4k"
Sep 17 20:51:37.372883 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:37.372821    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7eb327ea-1cee-4481-8533-423762e782e0-metrics-certs\") pod \"router-default-8b75c8b68-8wx4k\" (UID: \"7eb327ea-1cee-4481-8533-423762e782e0\") " pod="openshift-ingress/router-default-8b75c8b68-8wx4k"
Sep 17 20:51:37.373396 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:37.373380    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7eb327ea-1cee-4481-8533-423762e782e0-service-ca-bundle\") pod \"router-default-8b75c8b68-8wx4k\" (UID: \"7eb327ea-1cee-4481-8533-423762e782e0\") " pod="openshift-ingress/router-default-8b75c8b68-8wx4k"
Sep 17 20:51:37.374862 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:37.374832    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7eb327ea-1cee-4481-8533-423762e782e0-metrics-certs\") pod \"router-default-8b75c8b68-8wx4k\" (UID: \"7eb327ea-1cee-4481-8533-423762e782e0\") " pod="openshift-ingress/router-default-8b75c8b68-8wx4k"
Sep 17 20:51:37.473445 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:37.473406    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a95f9e39-2b33-434d-b6d0-807da6f8f4d2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-lm54d\" (UID: \"a95f9e39-2b33-434d-b6d0-807da6f8f4d2\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-lm54d"
Sep 17 20:51:37.476021 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:37.476001    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a95f9e39-2b33-434d-b6d0-807da6f8f4d2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-lm54d\" (UID: \"a95f9e39-2b33-434d-b6d0-807da6f8f4d2\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-lm54d"
Sep 17 20:51:37.566201 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:37.566155    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-8b75c8b68-8wx4k"
Sep 17 20:51:37.655957 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:37.655930    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-lm54d"
Sep 17 20:51:37.840454 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:37.840261    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-8b75c8b68-8wx4k"]
Sep 17 20:51:37.842691 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:51:37.842670    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eb327ea_1cee_4481_8533_423762e782e0.slice/crio-72eccbf875edebe0566216473808891baf074b98265cd525dd6e300db4b6bc21 WatchSource:0}: Error finding container 72eccbf875edebe0566216473808891baf074b98265cd525dd6e300db4b6bc21: Status 404 returned error can't find the container with id 72eccbf875edebe0566216473808891baf074b98265cd525dd6e300db4b6bc21
Sep 17 20:51:37.896435 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:37.896412    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-8b75c8b68-8wx4k" event={"ID":"7eb327ea-1cee-4481-8533-423762e782e0","Type":"ContainerStarted","Data":"a9429fa46e775b0f8614abccb86e4b980be54fe3806e96e80b89888e6be018fe"}
Sep 17 20:51:37.896524 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:37.896459    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-8b75c8b68-8wx4k" event={"ID":"7eb327ea-1cee-4481-8533-423762e782e0","Type":"ContainerStarted","Data":"72eccbf875edebe0566216473808891baf074b98265cd525dd6e300db4b6bc21"}
Sep 17 20:51:37.928213 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:37.928187    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-lm54d"]
Sep 17 20:51:37.931302 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:51:37.931277    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda95f9e39_2b33_434d_b6d0_807da6f8f4d2.slice/crio-855daa8e0d2c13d7c8022aa2804994b8333cd169143473d9f01d794cc370c65f WatchSource:0}: Error finding container 855daa8e0d2c13d7c8022aa2804994b8333cd169143473d9f01d794cc370c65f: Status 404 returned error can't find the container with id 855daa8e0d2c13d7c8022aa2804994b8333cd169143473d9f01d794cc370c65f
Sep 17 20:51:38.901588 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:38.901547    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-lm54d" event={"ID":"a95f9e39-2b33-434d-b6d0-807da6f8f4d2","Type":"ContainerStarted","Data":"855daa8e0d2c13d7c8022aa2804994b8333cd169143473d9f01d794cc370c65f"}
Sep 17 20:51:38.923435 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:38.923382    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-8b75c8b68-8wx4k" podStartSLOduration=33.923361027 podStartE2EDuration="33.923361027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:51:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:51:38.922054157 +0000 UTC m=+164.074498532" watchObservedRunningTime="2026-09-17 20:51:38.923361027 +0000 UTC m=+164.075805404"
Sep 17 20:51:39.566809 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:39.566785    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-8b75c8b68-8wx4k"
Sep 17 20:51:39.569320 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:39.569294    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-8b75c8b68-8wx4k"
Sep 17 20:51:39.827382 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:39.827302    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:39.905971 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:39.905932    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-lm54d" event={"ID":"a95f9e39-2b33-434d-b6d0-807da6f8f4d2","Type":"ContainerStarted","Data":"b1115e6efca332f668125198d92ea4ecef640147226c757880cb26068348dd22"}
Sep 17 20:51:39.906387 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:39.906220    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-8b75c8b68-8wx4k"
Sep 17 20:51:39.907335 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:39.907313    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-8b75c8b68-8wx4k"
Sep 17 20:51:39.924310 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:39.924072    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-lm54d" podStartSLOduration=33.345696033 podStartE2EDuration="34.92405392s" totalImagesPullingTime="1.578357887s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:51:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:51:39.923188405 +0000 UTC m=+165.075632784" watchObservedRunningTime="2026-09-17 20:51:39.92405392 +0000 UTC m=+165.076498297"
Sep 17 20:51:39.928839 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:39.928810    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:40.029154 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:40.029124    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:40.130719 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:40.130635    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:40.219716 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:40.219682    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:41.000500 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:41.000470    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fc616903-063e-4c1c-b98e-472c4be8203f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qsltg\" (UID: \"fc616903-063e-4c1c-b98e-472c4be8203f\") " pod="openshift-insights/insights-runtime-extractor-qsltg"
Sep 17 20:51:41.002818 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:41.002797    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fc616903-063e-4c1c-b98e-472c4be8203f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qsltg\" (UID: \"fc616903-063e-4c1c-b98e-472c4be8203f\") " pod="openshift-insights/insights-runtime-extractor-qsltg"
Sep 17 20:51:41.202669 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:41.202639    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qsltg"
Sep 17 20:51:41.474681 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:41.474636    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qsltg"]
Sep 17 20:51:41.477498 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:51:41.477467    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc616903_063e_4c1c_b98e_472c4be8203f.slice/crio-0a1f473b1ee582e60007fb5541e192f2ec0b0dbde6d363a6d577215ca53b3081 WatchSource:0}: Error finding container 0a1f473b1ee582e60007fb5541e192f2ec0b0dbde6d363a6d577215ca53b3081: Status 404 returned error can't find the container with id 0a1f473b1ee582e60007fb5541e192f2ec0b0dbde6d363a6d577215ca53b3081
Sep 17 20:51:41.912843 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:41.912771    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qsltg" event={"ID":"fc616903-063e-4c1c-b98e-472c4be8203f","Type":"ContainerStarted","Data":"11bf764a0917010f1bc64e55d8cb7efdeeca0e00b81062758447a2aca8dd1ab0"}
Sep 17 20:51:41.912843 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:41.912824    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qsltg" event={"ID":"fc616903-063e-4c1c-b98e-472c4be8203f","Type":"ContainerStarted","Data":"0a1f473b1ee582e60007fb5541e192f2ec0b0dbde6d363a6d577215ca53b3081"}
Sep 17 20:51:42.918543 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:42.918501    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qsltg" event={"ID":"fc616903-063e-4c1c-b98e-472c4be8203f","Type":"ContainerStarted","Data":"763e61b122c3c3b0ba606d20d7a2f3fdc8fb1c9871e142e770933ccb34ca0593"}
Sep 17 20:51:44.927441 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:44.927406    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qsltg" event={"ID":"fc616903-063e-4c1c-b98e-472c4be8203f","Type":"ContainerStarted","Data":"ea381e4b1515b487c12f33c6f919ae7af9f87f29243416958747c4242024af61"}
Sep 17 20:51:44.946923 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:44.946880    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-qsltg" podStartSLOduration=33.672517701 podStartE2EDuration="35.946865221s" totalImagesPullingTime="2.27434752s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:51:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:51:44.946129186 +0000 UTC m=+170.098573561" watchObservedRunningTime="2026-09-17 20:51:44.946865221 +0000 UTC m=+170.099309587"
Sep 17 20:51:45.351921 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:45.351892    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8b6dt"
Sep 17 20:51:46.975469 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:46.975440    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-gjz76"]
Sep 17 20:51:46.977586 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:46.977561    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-kdcrx"]
Sep 17 20:51:46.977699 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:46.977660    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-gjz76"
Sep 17 20:51:46.979648 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:46.979629    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-kdcrx"
Sep 17 20:51:46.988075 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:46.988054    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 17 20:51:46.988449 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:46.988427    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-dgfkn\""
Sep 17 20:51:46.988449 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:46.988449    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 17 20:51:46.988610 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:46.988472    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-gr2xt\""
Sep 17 20:51:47.000903 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:47.000772    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-kdcrx"]
Sep 17 20:51:47.002170 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:47.002149    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-gjz76"]
Sep 17 20:51:47.041774 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:47.041750    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk2jf\" (UniqueName: \"kubernetes.io/projected/fe419234-5829-4967-8b44-55eadbe489b3-kube-api-access-vk2jf\") pod \"network-check-source-7cbc45fddc-kdcrx\" (UID: \"fe419234-5829-4967-8b44-55eadbe489b3\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-kdcrx"
Sep 17 20:51:47.041922 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:47.041805    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f88bd94f-31d4-4c7a-a22d-0be24fae956e-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-gjz76\" (UID: \"f88bd94f-31d4-4c7a-a22d-0be24fae956e\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-gjz76"
Sep 17 20:51:47.041922 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:47.041885    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f88bd94f-31d4-4c7a-a22d-0be24fae956e-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-gjz76\" (UID: \"f88bd94f-31d4-4c7a-a22d-0be24fae956e\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-gjz76"
Sep 17 20:51:47.142583 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:47.142555    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk2jf\" (UniqueName: \"kubernetes.io/projected/fe419234-5829-4967-8b44-55eadbe489b3-kube-api-access-vk2jf\") pod \"network-check-source-7cbc45fddc-kdcrx\" (UID: \"fe419234-5829-4967-8b44-55eadbe489b3\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-kdcrx"
Sep 17 20:51:47.142757 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:47.142598    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f88bd94f-31d4-4c7a-a22d-0be24fae956e-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-gjz76\" (UID: \"f88bd94f-31d4-4c7a-a22d-0be24fae956e\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-gjz76"
Sep 17 20:51:47.142757 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:47.142722    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f88bd94f-31d4-4c7a-a22d-0be24fae956e-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-gjz76\" (UID: \"f88bd94f-31d4-4c7a-a22d-0be24fae956e\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-gjz76"
Sep 17 20:51:47.143355 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:47.143337    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f88bd94f-31d4-4c7a-a22d-0be24fae956e-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-gjz76\" (UID: \"f88bd94f-31d4-4c7a-a22d-0be24fae956e\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-gjz76"
Sep 17 20:51:47.144881 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:47.144865    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f88bd94f-31d4-4c7a-a22d-0be24fae956e-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-gjz76\" (UID: \"f88bd94f-31d4-4c7a-a22d-0be24fae956e\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-gjz76"
Sep 17 20:51:47.166121 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:47.166092    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk2jf\" (UniqueName: \"kubernetes.io/projected/fe419234-5829-4967-8b44-55eadbe489b3-kube-api-access-vk2jf\") pod \"network-check-source-7cbc45fddc-kdcrx\" (UID: \"fe419234-5829-4967-8b44-55eadbe489b3\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-kdcrx"
Sep 17 20:51:47.288121 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:47.288057    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-gjz76"
Sep 17 20:51:47.293014 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:47.292996    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-kdcrx"
Sep 17 20:51:47.306176 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:47.306149    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:47.351595 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:47.351567    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-d7m9q"
Sep 17 20:51:47.352248 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:47.352226    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c4b45cb74-s4gzt"
Sep 17 20:51:47.358456 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:47.358349    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2jjw4\""
Sep 17 20:51:47.358456 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:47.358407    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-wszs2\""
Sep 17 20:51:47.362245 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:47.362129    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-d7m9q"
Sep 17 20:51:47.363352 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:47.363335    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c4b45cb74-s4gzt"
Sep 17 20:51:47.444217 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:47.444192    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:47.564026 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:47.563997    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-gjz76"]
Sep 17 20:51:47.566322 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:51:47.566297    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf88bd94f_31d4_4c7a_a22d_0be24fae956e.slice/crio-731e1691d8ae4033843ac3982fc74cffad758d9e52936d404513e3f9a55f5c89 WatchSource:0}: Error finding container 731e1691d8ae4033843ac3982fc74cffad758d9e52936d404513e3f9a55f5c89: Status 404 returned error can't find the container with id 731e1691d8ae4033843ac3982fc74cffad758d9e52936d404513e3f9a55f5c89
Sep 17 20:51:47.568292 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:47.568211    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-kdcrx"]
Sep 17 20:51:47.665443 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:47.665414    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-d7m9q"]
Sep 17 20:51:47.670635 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:51:47.670608    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3afbe98f_5052_44e2_9d4b_da4bd2d30c3a.slice/crio-ecd53cefa30ec5f1150b4df4dec0c4eb7b2acbfd3bd7f380148e58e70a707740 WatchSource:0}: Error finding container ecd53cefa30ec5f1150b4df4dec0c4eb7b2acbfd3bd7f380148e58e70a707740: Status 404 returned error can't find the container with id ecd53cefa30ec5f1150b4df4dec0c4eb7b2acbfd3bd7f380148e58e70a707740
Sep 17 20:51:47.686050 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:47.686029    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6c4b45cb74-s4gzt"]
Sep 17 20:51:47.689256 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:51:47.689228    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dda82c0_e203_412d_8d70_bffaf6285281.slice/crio-5a02b7a1c53752968c30ae7225648eb460fe279946de56cd13fb476e0b3a174d WatchSource:0}: Error finding container 5a02b7a1c53752968c30ae7225648eb460fe279946de56cd13fb476e0b3a174d: Status 404 returned error can't find the container with id 5a02b7a1c53752968c30ae7225648eb460fe279946de56cd13fb476e0b3a174d
Sep 17 20:51:47.942279 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:47.942147    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-kdcrx" event={"ID":"fe419234-5829-4967-8b44-55eadbe489b3","Type":"ContainerStarted","Data":"b5c7ddd2d6abfa711874102bc875697defbedf9b52c991c3e290397e559da9be"}
Sep 17 20:51:47.942279 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:47.942219    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-kdcrx" event={"ID":"fe419234-5829-4967-8b44-55eadbe489b3","Type":"ContainerStarted","Data":"76f0b8c952252d1a77be295f5713ad01e61d72ac5f22218c93eebe161bb00efe"}
Sep 17 20:51:47.944093 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:47.944066    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c4b45cb74-s4gzt" event={"ID":"9dda82c0-e203-412d-8d70-bffaf6285281","Type":"ContainerStarted","Data":"51e9cd5e9a27a838c0ab939d252ced75e9aebe1a9813b8025bc87eadd82d508f"}
Sep 17 20:51:47.944216 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:47.944112    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c4b45cb74-s4gzt" event={"ID":"9dda82c0-e203-412d-8d70-bffaf6285281","Type":"ContainerStarted","Data":"5a02b7a1c53752968c30ae7225648eb460fe279946de56cd13fb476e0b3a174d"}
Sep 17 20:51:47.944216 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:47.944139    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6c4b45cb74-s4gzt"
Sep 17 20:51:47.945438 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:47.945410    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d7m9q" event={"ID":"3afbe98f-5052-44e2-9d4b-da4bd2d30c3a","Type":"ContainerStarted","Data":"ecd53cefa30ec5f1150b4df4dec0c4eb7b2acbfd3bd7f380148e58e70a707740"}
Sep 17 20:51:47.946617 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:47.946586    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-gjz76" event={"ID":"f88bd94f-31d4-4c7a-a22d-0be24fae956e","Type":"ContainerStarted","Data":"731e1691d8ae4033843ac3982fc74cffad758d9e52936d404513e3f9a55f5c89"}
Sep 17 20:51:47.959013 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:47.958968    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-kdcrx" podStartSLOduration=1.958955701 podStartE2EDuration="1.958955701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:51:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:51:47.957327084 +0000 UTC m=+173.109771461" watchObservedRunningTime="2026-09-17 20:51:47.958955701 +0000 UTC m=+173.111400068"
Sep 17 20:51:47.979287 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:47.979227    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6c4b45cb74-s4gzt" podStartSLOduration=171.979206796 podStartE2EDuration="2m51.979206796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:48:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:51:47.977239007 +0000 UTC m=+173.129683385" watchObservedRunningTime="2026-09-17 20:51:47.979206796 +0000 UTC m=+173.131651171"
Sep 17 20:51:48.951925 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:48.951867    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-gjz76" event={"ID":"f88bd94f-31d4-4c7a-a22d-0be24fae956e","Type":"ContainerStarted","Data":"da63c151d141c547c53b281d91858c990f1ae1c04a0b5ac97f05f912c1fd5387"}
Sep 17 20:51:48.972865 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:48.972800    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6446c8d75d-gjz76" podStartSLOduration=1.848367753 podStartE2EDuration="2.972780583s" totalImagesPullingTime="1.12441283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:51:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:51:48.970829837 +0000 UTC m=+174.123274215" watchObservedRunningTime="2026-09-17 20:51:48.972780583 +0000 UTC m=+174.125224974"
Sep 17 20:51:49.105464 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:49.105424    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:49.177264 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:49.177230    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:49.956789 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:49.956699    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d7m9q" event={"ID":"3afbe98f-5052-44e2-9d4b-da4bd2d30c3a","Type":"ContainerStarted","Data":"40d8fcdaf0410767f433ee4b8d32d6ec33e11d4c1b13f3264e61fc89b9354731"}
Sep 17 20:51:49.956789 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:49.956759    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d7m9q" event={"ID":"3afbe98f-5052-44e2-9d4b-da4bd2d30c3a","Type":"ContainerStarted","Data":"380df45657ef7ab69e0021f40e052f2f08894839af3531b97b2c3ea98579889c"}
Sep 17 20:51:49.956789 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:49.956783    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-d7m9q"
Sep 17 20:51:49.975579 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:49.975533    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-d7m9q" podStartSLOduration=138.940041232 podStartE2EDuration="2m20.975518575s" totalImagesPullingTime="2.035477343s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:49:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:51:49.973927684 +0000 UTC m=+175.126372061" watchObservedRunningTime="2026-09-17 20:51:49.975518575 +0000 UTC m=+175.127962962"
Sep 17 20:51:50.062505 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:50.062471    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:50.125047 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:50.125019    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-pp9dx"]
Sep 17 20:51:50.128030 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:50.128014    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-pp9dx"
Sep 17 20:51:50.130691 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:50.130674    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 17 20:51:50.130800 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:50.130711    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 17 20:51:50.130871 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:50.130843    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 20:51:50.131086 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:50.131073    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-qcsjx\""
Sep 17 20:51:50.140657 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:50.140627    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-pp9dx"]
Sep 17 20:51:50.160721 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:50.160685    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:50.168701 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:50.168679    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ecd1e7d2-5f90-44ea-a0d4-a8a3c14e90d5-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-pp9dx\" (UID: \"ecd1e7d2-5f90-44ea-a0d4-a8a3c14e90d5\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-pp9dx"
Sep 17 20:51:50.168769 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:50.168717    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zznfk\" (UniqueName: \"kubernetes.io/projected/ecd1e7d2-5f90-44ea-a0d4-a8a3c14e90d5-kube-api-access-zznfk\") pod \"prometheus-operator-7bd5d56d6-pp9dx\" (UID: \"ecd1e7d2-5f90-44ea-a0d4-a8a3c14e90d5\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-pp9dx"
Sep 17 20:51:50.168769 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:50.168751    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ecd1e7d2-5f90-44ea-a0d4-a8a3c14e90d5-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-pp9dx\" (UID: \"ecd1e7d2-5f90-44ea-a0d4-a8a3c14e90d5\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-pp9dx"
Sep 17 20:51:50.168843 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:50.168785    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ecd1e7d2-5f90-44ea-a0d4-a8a3c14e90d5-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-pp9dx\" (UID: \"ecd1e7d2-5f90-44ea-a0d4-a8a3c14e90d5\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-pp9dx"
Sep 17 20:51:50.258681 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:50.258594    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:50.269601 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:50.269576    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ecd1e7d2-5f90-44ea-a0d4-a8a3c14e90d5-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-pp9dx\" (UID: \"ecd1e7d2-5f90-44ea-a0d4-a8a3c14e90d5\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-pp9dx"
Sep 17 20:51:50.269715 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:50.269628    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ecd1e7d2-5f90-44ea-a0d4-a8a3c14e90d5-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-pp9dx\" (UID: \"ecd1e7d2-5f90-44ea-a0d4-a8a3c14e90d5\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-pp9dx"
Sep 17 20:51:50.269715 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:50.269685    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ecd1e7d2-5f90-44ea-a0d4-a8a3c14e90d5-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-pp9dx\" (UID: \"ecd1e7d2-5f90-44ea-a0d4-a8a3c14e90d5\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-pp9dx"
Sep 17 20:51:50.269828 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:50.269730    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zznfk\" (UniqueName: \"kubernetes.io/projected/ecd1e7d2-5f90-44ea-a0d4-a8a3c14e90d5-kube-api-access-zznfk\") pod \"prometheus-operator-7bd5d56d6-pp9dx\" (UID: \"ecd1e7d2-5f90-44ea-a0d4-a8a3c14e90d5\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-pp9dx"
Sep 17 20:51:50.270287 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:50.270268    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ecd1e7d2-5f90-44ea-a0d4-a8a3c14e90d5-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-pp9dx\" (UID: \"ecd1e7d2-5f90-44ea-a0d4-a8a3c14e90d5\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-pp9dx"
Sep 17 20:51:50.272098 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:50.272069    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ecd1e7d2-5f90-44ea-a0d4-a8a3c14e90d5-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-pp9dx\" (UID: \"ecd1e7d2-5f90-44ea-a0d4-a8a3c14e90d5\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-pp9dx"
Sep 17 20:51:50.272196 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:50.272181    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ecd1e7d2-5f90-44ea-a0d4-a8a3c14e90d5-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-pp9dx\" (UID: \"ecd1e7d2-5f90-44ea-a0d4-a8a3c14e90d5\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-pp9dx"
Sep 17 20:51:50.280650 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:50.280625    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zznfk\" (UniqueName: \"kubernetes.io/projected/ecd1e7d2-5f90-44ea-a0d4-a8a3c14e90d5-kube-api-access-zznfk\") pod \"prometheus-operator-7bd5d56d6-pp9dx\" (UID: \"ecd1e7d2-5f90-44ea-a0d4-a8a3c14e90d5\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-pp9dx"
Sep 17 20:51:50.436576 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:50.436522    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-pp9dx"
Sep 17 20:51:50.707003 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:50.706975    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-pp9dx"]
Sep 17 20:51:50.710001 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:51:50.709976    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecd1e7d2_5f90_44ea_a0d4_a8a3c14e90d5.slice/crio-7dfe9c6f0d3611e265ac15b0a8bc6fdcded0592788e5608dd334419342cae23b WatchSource:0}: Error finding container 7dfe9c6f0d3611e265ac15b0a8bc6fdcded0592788e5608dd334419342cae23b: Status 404 returned error can't find the container with id 7dfe9c6f0d3611e265ac15b0a8bc6fdcded0592788e5608dd334419342cae23b
Sep 17 20:51:50.960663 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:50.960573    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-pp9dx" event={"ID":"ecd1e7d2-5f90-44ea-a0d4-a8a3c14e90d5","Type":"ContainerStarted","Data":"7dfe9c6f0d3611e265ac15b0a8bc6fdcded0592788e5608dd334419342cae23b"}
Sep 17 20:51:51.966766 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:51.966730    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-pp9dx" event={"ID":"ecd1e7d2-5f90-44ea-a0d4-a8a3c14e90d5","Type":"ContainerStarted","Data":"a92333e2015f843a486277df52ae9daf6900b1aecf159a4f6deb850df23b01f7"}
Sep 17 20:51:51.967150 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:51.966783    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-pp9dx" event={"ID":"ecd1e7d2-5f90-44ea-a0d4-a8a3c14e90d5","Type":"ContainerStarted","Data":"0ab92b6db335f038b57c01bb56355847145b91915a5003da43f070e0b835cae7"}
Sep 17 20:51:51.988449 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:51.988392    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-pp9dx" podStartSLOduration=0.905796128 podStartE2EDuration="1.988372919s" totalImagesPullingTime="1.082576791s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:51:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:51:51.987877089 +0000 UTC m=+177.140321467" watchObservedRunningTime="2026-09-17 20:51:51.988372919 +0000 UTC m=+177.140817295"
Sep 17 20:51:52.124711 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:52.124629    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:53.272916 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:53.272883    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:53.363912 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:53.363878    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:53.566338 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:53.566255    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:53.633878 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:53.633833    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-9gdxc"]
Sep 17 20:51:53.638735 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:53.638718    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-9gdxc"
Sep 17 20:51:53.644367 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:53.644343    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 17 20:51:53.645146 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:53.645124    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-x8dhz\""
Sep 17 20:51:53.645238 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:53.645124    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 17 20:51:53.647838 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:53.647822    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 17 20:51:53.660579 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:53.660558    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-9gdxc"]
Sep 17 20:51:53.668896 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:53.668877    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-kb9r5"]
Sep 17 20:51:53.672243 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:53.672227    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kb9r5"
Sep 17 20:51:53.676593 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:53.676574    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 20:51:53.676931 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:53.676913    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 20:51:53.677381 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:53.677365    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-8f996\""
Sep 17 20:51:53.677914 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:53.677893    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 20:51:53.699495 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:53.699473    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0205bdac-de0b-423b-a6d5-e515f40473f1-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-9gdxc\" (UID: \"0205bdac-de0b-423b-a6d5-e515f40473f1\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-9gdxc"
Sep 17 20:51:53.699581 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:53.699501    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0205bdac-de0b-423b-a6d5-e515f40473f1-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-9gdxc\" (UID: \"0205bdac-de0b-423b-a6d5-e515f40473f1\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-9gdxc"
Sep 17 20:51:53.699581 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:53.699542    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqc9z\" (UniqueName: \"kubernetes.io/projected/0205bdac-de0b-423b-a6d5-e515f40473f1-kube-api-access-nqc9z\") pod \"kube-state-metrics-84c885658c-9gdxc\" (UID: \"0205bdac-de0b-423b-a6d5-e515f40473f1\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-9gdxc"
Sep 17 20:51:53.699581 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:53.699567    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0205bdac-de0b-423b-a6d5-e515f40473f1-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-9gdxc\" (UID: \"0205bdac-de0b-423b-a6d5-e515f40473f1\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-9gdxc"
Sep 17 20:51:53.699676 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:53.699598    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0205bdac-de0b-423b-a6d5-e515f40473f1-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-9gdxc\" (UID: \"0205bdac-de0b-423b-a6d5-e515f40473f1\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-9gdxc"
Sep 17 20:51:53.699708 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:53.699668    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0205bdac-de0b-423b-a6d5-e515f40473f1-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-9gdxc\" (UID: \"0205bdac-de0b-423b-a6d5-e515f40473f1\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-9gdxc"
Sep 17 20:51:53.722589 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:53.722564    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:53.800179 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:53.800153    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/80762970-c96e-4ab0-b6e8-c9c7107ed1fc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kb9r5\" (UID: \"80762970-c96e-4ab0-b6e8-c9c7107ed1fc\") " pod="openshift-monitoring/node-exporter-kb9r5"
Sep 17 20:51:53.800261 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:53.800186    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/80762970-c96e-4ab0-b6e8-c9c7107ed1fc-root\") pod \"node-exporter-kb9r5\" (UID: \"80762970-c96e-4ab0-b6e8-c9c7107ed1fc\") " pod="openshift-monitoring/node-exporter-kb9r5"
Sep 17 20:51:53.800261 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:53.800210    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/80762970-c96e-4ab0-b6e8-c9c7107ed1fc-metrics-client-ca\") pod \"node-exporter-kb9r5\" (UID: \"80762970-c96e-4ab0-b6e8-c9c7107ed1fc\") " pod="openshift-monitoring/node-exporter-kb9r5"
Sep 17 20:51:53.800325 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:53.800266    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0205bdac-de0b-423b-a6d5-e515f40473f1-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-9gdxc\" (UID: \"0205bdac-de0b-423b-a6d5-e515f40473f1\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-9gdxc"
Sep 17 20:51:53.800325 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:53.800296    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0205bdac-de0b-423b-a6d5-e515f40473f1-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-9gdxc\" (UID: \"0205bdac-de0b-423b-a6d5-e515f40473f1\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-9gdxc"
Sep 17 20:51:53.800391 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:53.800329    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqc9z\" (UniqueName: \"kubernetes.io/projected/0205bdac-de0b-423b-a6d5-e515f40473f1-kube-api-access-nqc9z\") pod \"kube-state-metrics-84c885658c-9gdxc\" (UID: \"0205bdac-de0b-423b-a6d5-e515f40473f1\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-9gdxc"
Sep 17 20:51:53.800391 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:53.800363    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/80762970-c96e-4ab0-b6e8-c9c7107ed1fc-node-exporter-textfile\") pod \"node-exporter-kb9r5\" (UID: \"80762970-c96e-4ab0-b6e8-c9c7107ed1fc\") " pod="openshift-monitoring/node-exporter-kb9r5"
Sep 17 20:51:53.800391 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:53.800381    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0205bdac-de0b-423b-a6d5-e515f40473f1-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-9gdxc\" (UID: \"0205bdac-de0b-423b-a6d5-e515f40473f1\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-9gdxc"
Sep 17 20:51:53.800502 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:53.800398    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg7c5\" (UniqueName: \"kubernetes.io/projected/80762970-c96e-4ab0-b6e8-c9c7107ed1fc-kube-api-access-fg7c5\") pod \"node-exporter-kb9r5\" (UID: \"80762970-c96e-4ab0-b6e8-c9c7107ed1fc\") " pod="openshift-monitoring/node-exporter-kb9r5"
Sep 17 20:51:53.800502 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:53.800413    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/80762970-c96e-4ab0-b6e8-c9c7107ed1fc-sys\") pod \"node-exporter-kb9r5\" (UID: \"80762970-c96e-4ab0-b6e8-c9c7107ed1fc\") " pod="openshift-monitoring/node-exporter-kb9r5"
Sep 17 20:51:53.800502 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:53.800453    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/80762970-c96e-4ab0-b6e8-c9c7107ed1fc-node-exporter-accelerators-collector-config\") pod \"node-exporter-kb9r5\" (UID: \"80762970-c96e-4ab0-b6e8-c9c7107ed1fc\") " pod="openshift-monitoring/node-exporter-kb9r5"
Sep 17 20:51:53.800638 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:53.800577    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0205bdac-de0b-423b-a6d5-e515f40473f1-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-9gdxc\" (UID: \"0205bdac-de0b-423b-a6d5-e515f40473f1\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-9gdxc"
Sep 17 20:51:53.800742 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:53.800725    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/80762970-c96e-4ab0-b6e8-c9c7107ed1fc-node-exporter-tls\") pod \"node-exporter-kb9r5\" (UID: \"80762970-c96e-4ab0-b6e8-c9c7107ed1fc\") " pod="openshift-monitoring/node-exporter-kb9r5"
Sep 17 20:51:53.800824 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:53.800756    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/80762970-c96e-4ab0-b6e8-c9c7107ed1fc-node-exporter-wtmp\") pod \"node-exporter-kb9r5\" (UID: \"80762970-c96e-4ab0-b6e8-c9c7107ed1fc\") " pod="openshift-monitoring/node-exporter-kb9r5"
Sep 17 20:51:53.800824 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:53.800794    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0205bdac-de0b-423b-a6d5-e515f40473f1-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-9gdxc\" (UID: \"0205bdac-de0b-423b-a6d5-e515f40473f1\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-9gdxc"
Sep 17 20:51:53.801054 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:53.801031    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0205bdac-de0b-423b-a6d5-e515f40473f1-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-9gdxc\" (UID: \"0205bdac-de0b-423b-a6d5-e515f40473f1\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-9gdxc"
Sep 17 20:51:53.801161 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:53.801109    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0205bdac-de0b-423b-a6d5-e515f40473f1-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-9gdxc\" (UID: \"0205bdac-de0b-423b-a6d5-e515f40473f1\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-9gdxc"
Sep 17 20:51:53.801224 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:53.801197    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0205bdac-de0b-423b-a6d5-e515f40473f1-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-9gdxc\" (UID: \"0205bdac-de0b-423b-a6d5-e515f40473f1\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-9gdxc"
Sep 17 20:51:53.803005 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:53.802981    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0205bdac-de0b-423b-a6d5-e515f40473f1-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-9gdxc\" (UID: \"0205bdac-de0b-423b-a6d5-e515f40473f1\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-9gdxc"
Sep 17 20:51:53.803088 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:53.802982    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0205bdac-de0b-423b-a6d5-e515f40473f1-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-9gdxc\" (UID: \"0205bdac-de0b-423b-a6d5-e515f40473f1\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-9gdxc"
Sep 17 20:51:53.817640 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:53.817581    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqc9z\" (UniqueName: \"kubernetes.io/projected/0205bdac-de0b-423b-a6d5-e515f40473f1-kube-api-access-nqc9z\") pod \"kube-state-metrics-84c885658c-9gdxc\" (UID: \"0205bdac-de0b-423b-a6d5-e515f40473f1\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-9gdxc"
Sep 17 20:51:53.854597 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:53.854575    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:53.901286 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:53.901263    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/80762970-c96e-4ab0-b6e8-c9c7107ed1fc-node-exporter-tls\") pod \"node-exporter-kb9r5\" (UID: \"80762970-c96e-4ab0-b6e8-c9c7107ed1fc\") " pod="openshift-monitoring/node-exporter-kb9r5"
Sep 17 20:51:53.901390 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:53.901294    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/80762970-c96e-4ab0-b6e8-c9c7107ed1fc-node-exporter-wtmp\") pod \"node-exporter-kb9r5\" (UID: \"80762970-c96e-4ab0-b6e8-c9c7107ed1fc\") " pod="openshift-monitoring/node-exporter-kb9r5"
Sep 17 20:51:53.901390 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:53.901345    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/80762970-c96e-4ab0-b6e8-c9c7107ed1fc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kb9r5\" (UID: \"80762970-c96e-4ab0-b6e8-c9c7107ed1fc\") " pod="openshift-monitoring/node-exporter-kb9r5"
Sep 17 20:51:53.901390 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:53.901378    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/80762970-c96e-4ab0-b6e8-c9c7107ed1fc-root\") pod \"node-exporter-kb9r5\" (UID: \"80762970-c96e-4ab0-b6e8-c9c7107ed1fc\") " pod="openshift-monitoring/node-exporter-kb9r5"
Sep 17 20:51:53.901541 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:53.901401    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/80762970-c96e-4ab0-b6e8-c9c7107ed1fc-metrics-client-ca\") pod \"node-exporter-kb9r5\" (UID: \"80762970-c96e-4ab0-b6e8-c9c7107ed1fc\") " pod="openshift-monitoring/node-exporter-kb9r5"
Sep 17 20:51:53.901541 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:53.901463    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/80762970-c96e-4ab0-b6e8-c9c7107ed1fc-node-exporter-textfile\") pod \"node-exporter-kb9r5\" (UID: \"80762970-c96e-4ab0-b6e8-c9c7107ed1fc\") " pod="openshift-monitoring/node-exporter-kb9r5"
Sep 17 20:51:53.901541 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:53.901492    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg7c5\" (UniqueName: \"kubernetes.io/projected/80762970-c96e-4ab0-b6e8-c9c7107ed1fc-kube-api-access-fg7c5\") pod \"node-exporter-kb9r5\" (UID: \"80762970-c96e-4ab0-b6e8-c9c7107ed1fc\") " pod="openshift-monitoring/node-exporter-kb9r5"
Sep 17 20:51:53.901541 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:53.901516    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/80762970-c96e-4ab0-b6e8-c9c7107ed1fc-sys\") pod \"node-exporter-kb9r5\" (UID: \"80762970-c96e-4ab0-b6e8-c9c7107ed1fc\") " pod="openshift-monitoring/node-exporter-kb9r5"
Sep 17 20:51:53.901760 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:53.901532    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/80762970-c96e-4ab0-b6e8-c9c7107ed1fc-node-exporter-wtmp\") pod \"node-exporter-kb9r5\" (UID: \"80762970-c96e-4ab0-b6e8-c9c7107ed1fc\") " pod="openshift-monitoring/node-exporter-kb9r5"
Sep 17 20:51:53.901760 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:53.901585    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/80762970-c96e-4ab0-b6e8-c9c7107ed1fc-sys\") pod \"node-exporter-kb9r5\" (UID: \"80762970-c96e-4ab0-b6e8-c9c7107ed1fc\") " pod="openshift-monitoring/node-exporter-kb9r5"
Sep 17 20:51:53.901760 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:53.901636    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/80762970-c96e-4ab0-b6e8-c9c7107ed1fc-root\") pod \"node-exporter-kb9r5\" (UID: \"80762970-c96e-4ab0-b6e8-c9c7107ed1fc\") " pod="openshift-monitoring/node-exporter-kb9r5"
Sep 17 20:51:53.901760 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:53.901684    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/80762970-c96e-4ab0-b6e8-c9c7107ed1fc-node-exporter-accelerators-collector-config\") pod \"node-exporter-kb9r5\" (UID: \"80762970-c96e-4ab0-b6e8-c9c7107ed1fc\") " pod="openshift-monitoring/node-exporter-kb9r5"
Sep 17 20:51:53.901950 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:53.901902    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/80762970-c96e-4ab0-b6e8-c9c7107ed1fc-node-exporter-textfile\") pod \"node-exporter-kb9r5\" (UID: \"80762970-c96e-4ab0-b6e8-c9c7107ed1fc\") " pod="openshift-monitoring/node-exporter-kb9r5"
Sep 17 20:51:53.902123 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:53.902101    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/80762970-c96e-4ab0-b6e8-c9c7107ed1fc-metrics-client-ca\") pod \"node-exporter-kb9r5\" (UID: \"80762970-c96e-4ab0-b6e8-c9c7107ed1fc\") " pod="openshift-monitoring/node-exporter-kb9r5"
Sep 17 20:51:53.902158 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:53.902134    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/80762970-c96e-4ab0-b6e8-c9c7107ed1fc-node-exporter-accelerators-collector-config\") pod \"node-exporter-kb9r5\" (UID: \"80762970-c96e-4ab0-b6e8-c9c7107ed1fc\") " pod="openshift-monitoring/node-exporter-kb9r5"
Sep 17 20:51:53.903460 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:53.903443    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/80762970-c96e-4ab0-b6e8-c9c7107ed1fc-node-exporter-tls\") pod \"node-exporter-kb9r5\" (UID: \"80762970-c96e-4ab0-b6e8-c9c7107ed1fc\") " pod="openshift-monitoring/node-exporter-kb9r5"
Sep 17 20:51:53.903545 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:53.903508    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/80762970-c96e-4ab0-b6e8-c9c7107ed1fc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kb9r5\" (UID: \"80762970-c96e-4ab0-b6e8-c9c7107ed1fc\") " pod="openshift-monitoring/node-exporter-kb9r5"
Sep 17 20:51:53.910737 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:53.910714    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg7c5\" (UniqueName: \"kubernetes.io/projected/80762970-c96e-4ab0-b6e8-c9c7107ed1fc-kube-api-access-fg7c5\") pod \"node-exporter-kb9r5\" (UID: \"80762970-c96e-4ab0-b6e8-c9c7107ed1fc\") " pod="openshift-monitoring/node-exporter-kb9r5"
Sep 17 20:51:53.947684 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:53.947659    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-9gdxc"
Sep 17 20:51:53.954661 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:53.954633    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:53.980574 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:53.980543    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kb9r5"
Sep 17 20:51:53.993599 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:51:53.993574    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80762970_c96e_4ab0_b6e8_c9c7107ed1fc.slice/crio-32dbb7c0d4a1367103ed8c122d99f2fc9cf170309155c2ad904a91f07e2bb4d3 WatchSource:0}: Error finding container 32dbb7c0d4a1367103ed8c122d99f2fc9cf170309155c2ad904a91f07e2bb4d3: Status 404 returned error can't find the container with id 32dbb7c0d4a1367103ed8c122d99f2fc9cf170309155c2ad904a91f07e2bb4d3
Sep 17 20:51:54.084184 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:54.084095    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:54.230692 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:54.230659    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-9gdxc"]
Sep 17 20:51:54.233841 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:51:54.233813    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0205bdac_de0b_423b_a6d5_e515f40473f1.slice/crio-a2edfe02e6ca35141731d2a7bde5a536257540208c4148208e8a0e026b43ffe0 WatchSource:0}: Error finding container a2edfe02e6ca35141731d2a7bde5a536257540208c4148208e8a0e026b43ffe0: Status 404 returned error can't find the container with id a2edfe02e6ca35141731d2a7bde5a536257540208c4148208e8a0e026b43ffe0
Sep 17 20:51:54.511517 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:54.511462    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:54.617789 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:54.617756    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 20:51:54.620677 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:54.620638    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:54.623456 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:54.623424    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:54.629729 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:54.627979    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 17 20:51:54.630396 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:54.630373    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 17 20:51:54.630501 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:54.630448    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-d4z5z\""
Sep 17 20:51:54.631139 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:54.630629    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 17 20:51:54.631139 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:54.630663    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 17 20:51:54.631139 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:54.630677    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 17 20:51:54.631139 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:54.630838    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 17 20:51:54.631139 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:54.630865    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 17 20:51:54.631139 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:54.630960    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 17 20:51:54.631139 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:54.631081    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 17 20:51:54.637817 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:54.637794    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 20:51:54.709368 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:54.709327    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/49b464f9-52b0-49b6-a3ff-a748723f65ee-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"49b464f9-52b0-49b6-a3ff-a748723f65ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:54.709530 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:54.709379    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/49b464f9-52b0-49b6-a3ff-a748723f65ee-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"49b464f9-52b0-49b6-a3ff-a748723f65ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:54.709530 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:54.709463    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/49b464f9-52b0-49b6-a3ff-a748723f65ee-config-out\") pod \"alertmanager-main-0\" (UID: \"49b464f9-52b0-49b6-a3ff-a748723f65ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:54.709530 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:54.709491    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/49b464f9-52b0-49b6-a3ff-a748723f65ee-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"49b464f9-52b0-49b6-a3ff-a748723f65ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:54.709828 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:54.709714    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/49b464f9-52b0-49b6-a3ff-a748723f65ee-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"49b464f9-52b0-49b6-a3ff-a748723f65ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:54.709828 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:54.709752    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/49b464f9-52b0-49b6-a3ff-a748723f65ee-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"49b464f9-52b0-49b6-a3ff-a748723f65ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:54.709828 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:54.709809    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/49b464f9-52b0-49b6-a3ff-a748723f65ee-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"49b464f9-52b0-49b6-a3ff-a748723f65ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:54.709992 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:54.709902    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p86bm\" (UniqueName: \"kubernetes.io/projected/49b464f9-52b0-49b6-a3ff-a748723f65ee-kube-api-access-p86bm\") pod \"alertmanager-main-0\" (UID: \"49b464f9-52b0-49b6-a3ff-a748723f65ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:54.709992 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:54.709973    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/49b464f9-52b0-49b6-a3ff-a748723f65ee-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"49b464f9-52b0-49b6-a3ff-a748723f65ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:54.710095 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:54.710000    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49b464f9-52b0-49b6-a3ff-a748723f65ee-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"49b464f9-52b0-49b6-a3ff-a748723f65ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:54.710095 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:54.710035    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/49b464f9-52b0-49b6-a3ff-a748723f65ee-web-config\") pod \"alertmanager-main-0\" (UID: \"49b464f9-52b0-49b6-a3ff-a748723f65ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:54.710095 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:54.710073    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/49b464f9-52b0-49b6-a3ff-a748723f65ee-config-volume\") pod \"alertmanager-main-0\" (UID: \"49b464f9-52b0-49b6-a3ff-a748723f65ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:54.710215 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:54.710097    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/49b464f9-52b0-49b6-a3ff-a748723f65ee-tls-assets\") pod \"alertmanager-main-0\" (UID: \"49b464f9-52b0-49b6-a3ff-a748723f65ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:54.810915 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:54.810684    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p86bm\" (UniqueName: \"kubernetes.io/projected/49b464f9-52b0-49b6-a3ff-a748723f65ee-kube-api-access-p86bm\") pod \"alertmanager-main-0\" (UID: \"49b464f9-52b0-49b6-a3ff-a748723f65ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:54.810915 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:54.810746    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/49b464f9-52b0-49b6-a3ff-a748723f65ee-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"49b464f9-52b0-49b6-a3ff-a748723f65ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:54.810915 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:54.810773    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49b464f9-52b0-49b6-a3ff-a748723f65ee-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"49b464f9-52b0-49b6-a3ff-a748723f65ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:54.810915 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:54.810803    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/49b464f9-52b0-49b6-a3ff-a748723f65ee-web-config\") pod \"alertmanager-main-0\" (UID: \"49b464f9-52b0-49b6-a3ff-a748723f65ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:54.810915 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:54.810833    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/49b464f9-52b0-49b6-a3ff-a748723f65ee-config-volume\") pod \"alertmanager-main-0\" (UID: \"49b464f9-52b0-49b6-a3ff-a748723f65ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:54.810915 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:54.810882    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/49b464f9-52b0-49b6-a3ff-a748723f65ee-tls-assets\") pod \"alertmanager-main-0\" (UID: \"49b464f9-52b0-49b6-a3ff-a748723f65ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:54.810915 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:54.810915    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/49b464f9-52b0-49b6-a3ff-a748723f65ee-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"49b464f9-52b0-49b6-a3ff-a748723f65ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:54.811354 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:54.810940    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/49b464f9-52b0-49b6-a3ff-a748723f65ee-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"49b464f9-52b0-49b6-a3ff-a748723f65ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:54.811354 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:54.810988    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/49b464f9-52b0-49b6-a3ff-a748723f65ee-config-out\") pod \"alertmanager-main-0\" (UID: \"49b464f9-52b0-49b6-a3ff-a748723f65ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:54.811354 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:54.811009    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/49b464f9-52b0-49b6-a3ff-a748723f65ee-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"49b464f9-52b0-49b6-a3ff-a748723f65ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:54.811354 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:54.811071    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/49b464f9-52b0-49b6-a3ff-a748723f65ee-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"49b464f9-52b0-49b6-a3ff-a748723f65ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:54.811354 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:54.811099    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/49b464f9-52b0-49b6-a3ff-a748723f65ee-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"49b464f9-52b0-49b6-a3ff-a748723f65ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:54.811354 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:54.811164    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/49b464f9-52b0-49b6-a3ff-a748723f65ee-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"49b464f9-52b0-49b6-a3ff-a748723f65ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:54.812010 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:51:54.811985    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/49b464f9-52b0-49b6-a3ff-a748723f65ee-alertmanager-trusted-ca-bundle podName:49b464f9-52b0-49b6-a3ff-a748723f65ee nodeName:}" failed. No retries permitted until 2026-09-17 20:51:55.311961301 +0000 UTC m=+180.464405669 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/49b464f9-52b0-49b6-a3ff-a748723f65ee-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "49b464f9-52b0-49b6-a3ff-a748723f65ee") : configmap references non-existent config key: ca-bundle.crt
Sep 17 20:51:54.812591 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:54.812558    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/49b464f9-52b0-49b6-a3ff-a748723f65ee-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"49b464f9-52b0-49b6-a3ff-a748723f65ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:54.812684 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:54.812599    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/49b464f9-52b0-49b6-a3ff-a748723f65ee-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"49b464f9-52b0-49b6-a3ff-a748723f65ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:54.815137 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:54.815060    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/49b464f9-52b0-49b6-a3ff-a748723f65ee-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"49b464f9-52b0-49b6-a3ff-a748723f65ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:54.816598 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:54.816550    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:54.817214 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:54.817183    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/49b464f9-52b0-49b6-a3ff-a748723f65ee-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"49b464f9-52b0-49b6-a3ff-a748723f65ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:54.819456 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:54.818835    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/49b464f9-52b0-49b6-a3ff-a748723f65ee-config-out\") pod \"alertmanager-main-0\" (UID: \"49b464f9-52b0-49b6-a3ff-a748723f65ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:54.821220 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:54.821057    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/49b464f9-52b0-49b6-a3ff-a748723f65ee-tls-assets\") pod \"alertmanager-main-0\" (UID: \"49b464f9-52b0-49b6-a3ff-a748723f65ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:54.822682 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:54.822660    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/49b464f9-52b0-49b6-a3ff-a748723f65ee-web-config\") pod \"alertmanager-main-0\" (UID: \"49b464f9-52b0-49b6-a3ff-a748723f65ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:54.823885 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:54.823826    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/49b464f9-52b0-49b6-a3ff-a748723f65ee-config-volume\") pod \"alertmanager-main-0\" (UID: \"49b464f9-52b0-49b6-a3ff-a748723f65ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:54.824141 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:54.824000    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/49b464f9-52b0-49b6-a3ff-a748723f65ee-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"49b464f9-52b0-49b6-a3ff-a748723f65ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:54.824141 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:54.824106    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/49b464f9-52b0-49b6-a3ff-a748723f65ee-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"49b464f9-52b0-49b6-a3ff-a748723f65ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:54.825148 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:54.825102    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p86bm\" (UniqueName: \"kubernetes.io/projected/49b464f9-52b0-49b6-a3ff-a748723f65ee-kube-api-access-p86bm\") pod \"alertmanager-main-0\" (UID: \"49b464f9-52b0-49b6-a3ff-a748723f65ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:54.825936 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:54.825911    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/49b464f9-52b0-49b6-a3ff-a748723f65ee-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"49b464f9-52b0-49b6-a3ff-a748723f65ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:54.914288 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:54.914168    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:54.984060 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:54.984028    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kb9r5" event={"ID":"80762970-c96e-4ab0-b6e8-c9c7107ed1fc","Type":"ContainerStarted","Data":"029af44e0f1e7834db668440295aad0a54de2f4c66a58a0164238f3256a19457"}
Sep 17 20:51:54.984190 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:54.984100    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kb9r5" event={"ID":"80762970-c96e-4ab0-b6e8-c9c7107ed1fc","Type":"ContainerStarted","Data":"32dbb7c0d4a1367103ed8c122d99f2fc9cf170309155c2ad904a91f07e2bb4d3"}
Sep 17 20:51:54.985287 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:54.985262    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-9gdxc" event={"ID":"0205bdac-de0b-423b-a6d5-e515f40473f1","Type":"ContainerStarted","Data":"a2edfe02e6ca35141731d2a7bde5a536257540208c4148208e8a0e026b43ffe0"}
Sep 17 20:51:55.314735 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:55.314703    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49b464f9-52b0-49b6-a3ff-a748723f65ee-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"49b464f9-52b0-49b6-a3ff-a748723f65ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:55.315665 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:55.315638    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49b464f9-52b0-49b6-a3ff-a748723f65ee-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"49b464f9-52b0-49b6-a3ff-a748723f65ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:55.542800 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:55.542771    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-d4z5z\""
Sep 17 20:51:55.551272 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:55.551230    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:55.842097 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:55.842065    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 20:51:55.845411 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:51:55.845386    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49b464f9_52b0_49b6_a3ff_a748723f65ee.slice/crio-090d24a84103d31c54b571844e81ecd8ef7760aa607ce43e9d4240517b073d0c WatchSource:0}: Error finding container 090d24a84103d31c54b571844e81ecd8ef7760aa607ce43e9d4240517b073d0c: Status 404 returned error can't find the container with id 090d24a84103d31c54b571844e81ecd8ef7760aa607ce43e9d4240517b073d0c
Sep 17 20:51:55.990916 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:55.990877    2575 generic.go:356] "Generic (PLEG): container finished" podID="80762970-c96e-4ab0-b6e8-c9c7107ed1fc" containerID="029af44e0f1e7834db668440295aad0a54de2f4c66a58a0164238f3256a19457" exitCode=0
Sep 17 20:51:55.991082 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:55.990912    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kb9r5" event={"ID":"80762970-c96e-4ab0-b6e8-c9c7107ed1fc","Type":"ContainerDied","Data":"029af44e0f1e7834db668440295aad0a54de2f4c66a58a0164238f3256a19457"}
Sep 17 20:51:55.993305 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:55.993283    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-9gdxc" event={"ID":"0205bdac-de0b-423b-a6d5-e515f40473f1","Type":"ContainerStarted","Data":"72f4708af6698a2c9180dca3663a7eff46a99e9f419e92401a78b90fa99abb33"}
Sep 17 20:51:55.993407 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:55.993341    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-9gdxc" event={"ID":"0205bdac-de0b-423b-a6d5-e515f40473f1","Type":"ContainerStarted","Data":"a05b592a093ae9d9497bc869d0e911416b8051c67bb58cb90417316558248a89"}
Sep 17 20:51:55.993407 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:55.993372    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-9gdxc" event={"ID":"0205bdac-de0b-423b-a6d5-e515f40473f1","Type":"ContainerStarted","Data":"c26ecc1691176e51ed27773e702ad5438d52a705e520d746b05eaec3b2b78b61"}
Sep 17 20:51:55.994742 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:55.994715    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"49b464f9-52b0-49b6-a3ff-a748723f65ee","Type":"ContainerStarted","Data":"090d24a84103d31c54b571844e81ecd8ef7760aa607ce43e9d4240517b073d0c"}
Sep 17 20:51:56.042303 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:56.042248    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-84c885658c-9gdxc" podStartSLOduration=1.920492688 podStartE2EDuration="3.042226658s" totalImagesPullingTime="1.12173397s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:51:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:51:56.040419426 +0000 UTC m=+181.192863803" watchObservedRunningTime="2026-09-17 20:51:56.042226658 +0000 UTC m=+181.194671028"
Sep 17 20:51:56.402208 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:56.402134    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:56.523450 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:56.523415    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:56.637570 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:56.637499    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:56.736126 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:56.736097    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:56.837326 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:56.837297    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:56.938183 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:56.938150    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:57.000230 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:57.000165    2575 generic.go:356] "Generic (PLEG): container finished" podID="49b464f9-52b0-49b6-a3ff-a748723f65ee" containerID="5f8f794442979f1ad6b90d6cc897660c0db739c62c86776ae476d8d1ac807f45" exitCode=0
Sep 17 20:51:57.000230 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:57.000200    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"49b464f9-52b0-49b6-a3ff-a748723f65ee","Type":"ContainerDied","Data":"5f8f794442979f1ad6b90d6cc897660c0db739c62c86776ae476d8d1ac807f45"}
Sep 17 20:51:57.005219 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:57.005195    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kb9r5" event={"ID":"80762970-c96e-4ab0-b6e8-c9c7107ed1fc","Type":"ContainerStarted","Data":"d56236cb8f239f04a17ac898e515f28d3ca5ab5ea036ff55d6f3a4dae34d498a"}
Sep 17 20:51:57.005310 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:57.005249    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kb9r5" event={"ID":"80762970-c96e-4ab0-b6e8-c9c7107ed1fc","Type":"ContainerStarted","Data":"56ac53e2486a08ea477cf5765ae06d8359be4aa5bc47d8ea0c8a1a74d35e21dc"}
Sep 17 20:51:57.051318 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:57.051270    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-kb9r5" podStartSLOduration=3.08211766 podStartE2EDuration="4.05125322s" totalImagesPullingTime="898.882842ms" totalInitContainerRuntime="70.252718ms" isStatefulPod=true podCreationTimestamp="2026-09-17 20:51:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:51:57.051170795 +0000 UTC m=+182.203615172" watchObservedRunningTime="2026-09-17 20:51:57.05125322 +0000 UTC m=+182.203697615"
Sep 17 20:51:57.165424 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:57.165389    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:57.232008 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:57.231970    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:57.377238 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:57.377146    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:57.775940 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:57.775904    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:57.972569 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:57.972526    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:58.096609 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:58.096515    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:58.149172 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:58.149149    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-67bbf8d54-844pn"]
Sep 17 20:51:58.160600 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:58.160579    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-67bbf8d54-844pn"
Sep 17 20:51:58.164211 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:58.164191    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-4z7dr\""
Sep 17 20:51:58.164315 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:58.164237    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 17 20:51:58.164445 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:58.164423    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-f3kt4vhispuvh\""
Sep 17 20:51:58.164796 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:58.164779    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 17 20:51:58.165160 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:58.165140    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 17 20:51:58.165239 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:58.165169    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 17 20:51:58.165386 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:58.165368    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-67bbf8d54-844pn"]
Sep 17 20:51:58.215634 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:58.215588    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:58.243125 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:58.243083    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54f14609-0f14-4c0f-87f7-aebeba7a2b0e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-67bbf8d54-844pn\" (UID: \"54f14609-0f14-4c0f-87f7-aebeba7a2b0e\") " pod="openshift-monitoring/metrics-server-67bbf8d54-844pn"
Sep 17 20:51:58.243224 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:58.243195    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/54f14609-0f14-4c0f-87f7-aebeba7a2b0e-secret-metrics-server-client-certs\") pod \"metrics-server-67bbf8d54-844pn\" (UID: \"54f14609-0f14-4c0f-87f7-aebeba7a2b0e\") " pod="openshift-monitoring/metrics-server-67bbf8d54-844pn"
Sep 17 20:51:58.243279 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:58.243241    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/54f14609-0f14-4c0f-87f7-aebeba7a2b0e-metrics-server-audit-profiles\") pod \"metrics-server-67bbf8d54-844pn\" (UID: \"54f14609-0f14-4c0f-87f7-aebeba7a2b0e\") " pod="openshift-monitoring/metrics-server-67bbf8d54-844pn"
Sep 17 20:51:58.243525 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:58.243366    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/54f14609-0f14-4c0f-87f7-aebeba7a2b0e-client-ca-bundle\") pod \"metrics-server-67bbf8d54-844pn\" (UID: \"54f14609-0f14-4c0f-87f7-aebeba7a2b0e\") " pod="openshift-monitoring/metrics-server-67bbf8d54-844pn"
Sep 17 20:51:58.243525 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:58.243439    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/54f14609-0f14-4c0f-87f7-aebeba7a2b0e-secret-metrics-server-tls\") pod \"metrics-server-67bbf8d54-844pn\" (UID: \"54f14609-0f14-4c0f-87f7-aebeba7a2b0e\") " pod="openshift-monitoring/metrics-server-67bbf8d54-844pn"
Sep 17 20:51:58.243525 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:58.243484    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/54f14609-0f14-4c0f-87f7-aebeba7a2b0e-audit-log\") pod \"metrics-server-67bbf8d54-844pn\" (UID: \"54f14609-0f14-4c0f-87f7-aebeba7a2b0e\") " pod="openshift-monitoring/metrics-server-67bbf8d54-844pn"
Sep 17 20:51:58.243525 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:58.243513    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cngql\" (UniqueName: \"kubernetes.io/projected/54f14609-0f14-4c0f-87f7-aebeba7a2b0e-kube-api-access-cngql\") pod \"metrics-server-67bbf8d54-844pn\" (UID: \"54f14609-0f14-4c0f-87f7-aebeba7a2b0e\") " pod="openshift-monitoring/metrics-server-67bbf8d54-844pn"
Sep 17 20:51:58.303173 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:58.303144    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-nrqnb"]
Sep 17 20:51:58.306203 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:58.306189    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-nrqnb"
Sep 17 20:51:58.309286 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:58.308168    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:58.309286 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:58.308776    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-qbdrc\""
Sep 17 20:51:58.309286 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:58.309049    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 17 20:51:58.315761 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:58.315740    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-nrqnb"]
Sep 17 20:51:58.344093 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:58.344061    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/54f14609-0f14-4c0f-87f7-aebeba7a2b0e-client-ca-bundle\") pod \"metrics-server-67bbf8d54-844pn\" (UID: \"54f14609-0f14-4c0f-87f7-aebeba7a2b0e\") " pod="openshift-monitoring/metrics-server-67bbf8d54-844pn"
Sep 17 20:51:58.344247 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:58.344229    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/54f14609-0f14-4c0f-87f7-aebeba7a2b0e-secret-metrics-server-tls\") pod \"metrics-server-67bbf8d54-844pn\" (UID: \"54f14609-0f14-4c0f-87f7-aebeba7a2b0e\") " pod="openshift-monitoring/metrics-server-67bbf8d54-844pn"
Sep 17 20:51:58.344320 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:58.344262    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/54f14609-0f14-4c0f-87f7-aebeba7a2b0e-audit-log\") pod \"metrics-server-67bbf8d54-844pn\" (UID: \"54f14609-0f14-4c0f-87f7-aebeba7a2b0e\") " pod="openshift-monitoring/metrics-server-67bbf8d54-844pn"
Sep 17 20:51:58.344320 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:58.344287    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cngql\" (UniqueName: \"kubernetes.io/projected/54f14609-0f14-4c0f-87f7-aebeba7a2b0e-kube-api-access-cngql\") pod \"metrics-server-67bbf8d54-844pn\" (UID: \"54f14609-0f14-4c0f-87f7-aebeba7a2b0e\") " pod="openshift-monitoring/metrics-server-67bbf8d54-844pn"
Sep 17 20:51:58.344422 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:58.344329    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54f14609-0f14-4c0f-87f7-aebeba7a2b0e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-67bbf8d54-844pn\" (UID: \"54f14609-0f14-4c0f-87f7-aebeba7a2b0e\") " pod="openshift-monitoring/metrics-server-67bbf8d54-844pn"
Sep 17 20:51:58.344422 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:58.344379    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/54f14609-0f14-4c0f-87f7-aebeba7a2b0e-secret-metrics-server-client-certs\") pod \"metrics-server-67bbf8d54-844pn\" (UID: \"54f14609-0f14-4c0f-87f7-aebeba7a2b0e\") " pod="openshift-monitoring/metrics-server-67bbf8d54-844pn"
Sep 17 20:51:58.344422 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:58.344415    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/54f14609-0f14-4c0f-87f7-aebeba7a2b0e-metrics-server-audit-profiles\") pod \"metrics-server-67bbf8d54-844pn\" (UID: \"54f14609-0f14-4c0f-87f7-aebeba7a2b0e\") " pod="openshift-monitoring/metrics-server-67bbf8d54-844pn"
Sep 17 20:51:58.345172 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:58.345143    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/54f14609-0f14-4c0f-87f7-aebeba7a2b0e-audit-log\") pod \"metrics-server-67bbf8d54-844pn\" (UID: \"54f14609-0f14-4c0f-87f7-aebeba7a2b0e\") " pod="openshift-monitoring/metrics-server-67bbf8d54-844pn"
Sep 17 20:51:58.345398 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:58.345372    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54f14609-0f14-4c0f-87f7-aebeba7a2b0e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-67bbf8d54-844pn\" (UID: \"54f14609-0f14-4c0f-87f7-aebeba7a2b0e\") " pod="openshift-monitoring/metrics-server-67bbf8d54-844pn"
Sep 17 20:51:58.345549 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:58.345526    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/54f14609-0f14-4c0f-87f7-aebeba7a2b0e-metrics-server-audit-profiles\") pod \"metrics-server-67bbf8d54-844pn\" (UID: \"54f14609-0f14-4c0f-87f7-aebeba7a2b0e\") " pod="openshift-monitoring/metrics-server-67bbf8d54-844pn"
Sep 17 20:51:58.347160 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:58.347089    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/54f14609-0f14-4c0f-87f7-aebeba7a2b0e-secret-metrics-server-tls\") pod \"metrics-server-67bbf8d54-844pn\" (UID: \"54f14609-0f14-4c0f-87f7-aebeba7a2b0e\") " pod="openshift-monitoring/metrics-server-67bbf8d54-844pn"
Sep 17 20:51:58.347252 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:58.347180    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/54f14609-0f14-4c0f-87f7-aebeba7a2b0e-client-ca-bundle\") pod \"metrics-server-67bbf8d54-844pn\" (UID: \"54f14609-0f14-4c0f-87f7-aebeba7a2b0e\") " pod="openshift-monitoring/metrics-server-67bbf8d54-844pn"
Sep 17 20:51:58.348479 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:58.348461    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/54f14609-0f14-4c0f-87f7-aebeba7a2b0e-secret-metrics-server-client-certs\") pod \"metrics-server-67bbf8d54-844pn\" (UID: \"54f14609-0f14-4c0f-87f7-aebeba7a2b0e\") " pod="openshift-monitoring/metrics-server-67bbf8d54-844pn"
Sep 17 20:51:58.355778 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:58.355749    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cngql\" (UniqueName: \"kubernetes.io/projected/54f14609-0f14-4c0f-87f7-aebeba7a2b0e-kube-api-access-cngql\") pod \"metrics-server-67bbf8d54-844pn\" (UID: \"54f14609-0f14-4c0f-87f7-aebeba7a2b0e\") " pod="openshift-monitoring/metrics-server-67bbf8d54-844pn"
Sep 17 20:51:58.444987 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:58.444959    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9dbbca3f-dbb6-40b3-9553-b710eddde112-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-nrqnb\" (UID: \"9dbbca3f-dbb6-40b3-9553-b710eddde112\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-nrqnb"
Sep 17 20:51:58.445110 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:58.445002    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfjkb\" (UniqueName: \"kubernetes.io/projected/9dbbca3f-dbb6-40b3-9553-b710eddde112-kube-api-access-hfjkb\") pod \"monitoring-plugin-5984459ff8-nrqnb\" (UID: \"9dbbca3f-dbb6-40b3-9553-b710eddde112\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-nrqnb"
Sep 17 20:51:58.476070 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:58.476044    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-67bbf8d54-844pn"
Sep 17 20:51:58.546158 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:58.546124    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9dbbca3f-dbb6-40b3-9553-b710eddde112-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-nrqnb\" (UID: \"9dbbca3f-dbb6-40b3-9553-b710eddde112\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-nrqnb"
Sep 17 20:51:58.546256 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:58.546183    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfjkb\" (UniqueName: \"kubernetes.io/projected/9dbbca3f-dbb6-40b3-9553-b710eddde112-kube-api-access-hfjkb\") pod \"monitoring-plugin-5984459ff8-nrqnb\" (UID: \"9dbbca3f-dbb6-40b3-9553-b710eddde112\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-nrqnb"
Sep 17 20:51:58.546308 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:51:58.546269    2575 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 17 20:51:58.546339 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:51:58.546331    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dbbca3f-dbb6-40b3-9553-b710eddde112-monitoring-plugin-cert podName:9dbbca3f-dbb6-40b3-9553-b710eddde112 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:59.046313999 +0000 UTC m=+184.198758353 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/9dbbca3f-dbb6-40b3-9553-b710eddde112-monitoring-plugin-cert") pod "monitoring-plugin-5984459ff8-nrqnb" (UID: "9dbbca3f-dbb6-40b3-9553-b710eddde112") : secret "monitoring-plugin-cert" not found
Sep 17 20:51:58.556783 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:58.556753    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfjkb\" (UniqueName: \"kubernetes.io/projected/9dbbca3f-dbb6-40b3-9553-b710eddde112-kube-api-access-hfjkb\") pod \"monitoring-plugin-5984459ff8-nrqnb\" (UID: \"9dbbca3f-dbb6-40b3-9553-b710eddde112\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-nrqnb"
Sep 17 20:51:58.634028 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:58.633944    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:51:58.767334 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:58.767312    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-67bbf8d54-844pn"]
Sep 17 20:51:58.768985 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:51:58.768959    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54f14609_0f14_4c0f_87f7_aebeba7a2b0e.slice/crio-d9f15c4880cfb3904553908b7f406533a0a560ffc50633c7b69538aa1bf8d184 WatchSource:0}: Error finding container d9f15c4880cfb3904553908b7f406533a0a560ffc50633c7b69538aa1bf8d184: Status 404 returned error can't find the container with id d9f15c4880cfb3904553908b7f406533a0a560ffc50633c7b69538aa1bf8d184
Sep 17 20:51:59.014031 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:59.014000    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-67bbf8d54-844pn" event={"ID":"54f14609-0f14-4c0f-87f7-aebeba7a2b0e","Type":"ContainerStarted","Data":"d9f15c4880cfb3904553908b7f406533a0a560ffc50633c7b69538aa1bf8d184"}
Sep 17 20:51:59.017142 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:59.017115    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"49b464f9-52b0-49b6-a3ff-a748723f65ee","Type":"ContainerStarted","Data":"84e65b0000c7155632fb0eee5568c1a583c876fb2b24c6560e6bf2286566398d"}
Sep 17 20:51:59.017265 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:59.017184    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"49b464f9-52b0-49b6-a3ff-a748723f65ee","Type":"ContainerStarted","Data":"8fe241f7e25e7e1bc5ce3b9f0c78d0d230f0f24637ed3b915c3862dfa1d2284e"}
Sep 17 20:51:59.017265 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:59.017210    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"49b464f9-52b0-49b6-a3ff-a748723f65ee","Type":"ContainerStarted","Data":"0b39670623a99267e9e72bfc5d3dd36dc89d6446518f71e949d855450624b303"}
Sep 17 20:51:59.017265 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:59.017235    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"49b464f9-52b0-49b6-a3ff-a748723f65ee","Type":"ContainerStarted","Data":"c373f74f20d20157e5369f7f2486198d1e026aa58b3d8b4067e63db567307cf5"}
Sep 17 20:51:59.017265 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:59.017260    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"49b464f9-52b0-49b6-a3ff-a748723f65ee","Type":"ContainerStarted","Data":"1760014afb019f8d1d6cd742ad4b68d3edfd366e776ad96b2e994eb0dd4cdbd2"}
Sep 17 20:51:59.049719 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:59.049695    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9dbbca3f-dbb6-40b3-9553-b710eddde112-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-nrqnb\" (UID: \"9dbbca3f-dbb6-40b3-9553-b710eddde112\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-nrqnb"
Sep 17 20:51:59.051947 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:59.051922    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9dbbca3f-dbb6-40b3-9553-b710eddde112-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-nrqnb\" (UID: \"9dbbca3f-dbb6-40b3-9553-b710eddde112\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-nrqnb"
Sep 17 20:51:59.219815 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:59.219790    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-nrqnb"
Sep 17 20:51:59.488116 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:59.488087    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-nrqnb"]
Sep 17 20:51:59.491035 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:51:59.491003    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dbbca3f_dbb6_40b3_9553_b710eddde112.slice/crio-0dc6af357ee1ef6697cccb07cba4e095bd9901faa2aa376c27c72be840dbee0c WatchSource:0}: Error finding container 0dc6af357ee1ef6697cccb07cba4e095bd9901faa2aa376c27c72be840dbee0c: Status 404 returned error can't find the container with id 0dc6af357ee1ef6697cccb07cba4e095bd9901faa2aa376c27c72be840dbee0c
Sep 17 20:51:59.965246 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:51:59.964345    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-d7m9q"
Sep 17 20:52:00.030039 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:00.029973    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"49b464f9-52b0-49b6-a3ff-a748723f65ee","Type":"ContainerStarted","Data":"f50c083b13d615a658b051ec57092cc96da60085b4224b8f2250fb270406a2b9"}
Sep 17 20:52:00.037266 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:00.035764    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-nrqnb" event={"ID":"9dbbca3f-dbb6-40b3-9553-b710eddde112","Type":"ContainerStarted","Data":"0dc6af357ee1ef6697cccb07cba4e095bd9901faa2aa376c27c72be840dbee0c"}
Sep 17 20:52:00.050246 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:00.050211    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:52:00.090969 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:00.090894    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.424615741 podStartE2EDuration="6.090869562s" totalImagesPullingTime="2.648677248s" totalInitContainerRuntime="17.576573ms" isStatefulPod=false podCreationTimestamp="2026-09-17 20:51:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:52:00.088959742 +0000 UTC m=+185.241404112" watchObservedRunningTime="2026-09-17 20:52:00.090869562 +0000 UTC m=+185.243313957"
Sep 17 20:52:01.040995 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:01.040883    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-nrqnb" event={"ID":"9dbbca3f-dbb6-40b3-9553-b710eddde112","Type":"ContainerStarted","Data":"b237a37931be2121885e2c28ddfd5289dd21f784b016a7c57a1eda2c621080a1"}
Sep 17 20:52:01.041426 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:01.041121    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-nrqnb"
Sep 17 20:52:01.042812 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:01.042776    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-67bbf8d54-844pn" event={"ID":"54f14609-0f14-4c0f-87f7-aebeba7a2b0e","Type":"ContainerStarted","Data":"25bec1b892a6716ae57198165aa79c791985325cd7325b1ce3c86c3415d7de51"}
Sep 17 20:52:01.046065 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:01.046045    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-nrqnb"
Sep 17 20:52:01.063940 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:01.063899    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5984459ff8-nrqnb" podStartSLOduration=1.7737741169999999 podStartE2EDuration="3.063886419s" totalImagesPullingTime="1.290112302s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:51:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:52:01.063669146 +0000 UTC m=+186.216113523" watchObservedRunningTime="2026-09-17 20:52:01.063886419 +0000 UTC m=+186.216330795"
Sep 17 20:52:01.101660 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:01.101610    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-67bbf8d54-844pn" podStartSLOduration=1.091687806 podStartE2EDuration="3.10159458s" totalImagesPullingTime="2.009906774s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:51:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:52:01.099325062 +0000 UTC m=+186.251769439" watchObservedRunningTime="2026-09-17 20:52:01.10159458 +0000 UTC m=+186.254038955"
Sep 17 20:52:07.367686 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:07.367652    2575 patch_prober.go:28] interesting pod/image-registry-6c4b45cb74-s4gzt container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 17 20:52:07.368180 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:07.367712    2575 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6c4b45cb74-s4gzt" podUID="9dda82c0-e203-412d-8d70-bffaf6285281" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 20:52:08.956833 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:08.956802    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6c4b45cb74-s4gzt"
Sep 17 20:52:09.124344 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:09.124308    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:52:09.223574 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:09.223478    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:52:09.388616 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:09.388581    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6c4b45cb74-s4gzt"]
Sep 17 20:52:09.474575 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:09.474489    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:52:09.574488 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:09.574459    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:52:09.674374 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:09.674324    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:52:12.603245 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:12.603210    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:52:12.705105 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:12.705067    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:52:12.804441 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:12.804406    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:52:14.098412 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:14.098380    2575 generic.go:356] "Generic (PLEG): container finished" podID="7e30e07a-26a2-4051-9dba-6170acb64eba" containerID="5f18a47765127888a6fb3cec4c9bbca017b4234700777938cb1199756c277bad" exitCode=0
Sep 17 20:52:14.098837 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:14.098426    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-jknl5" event={"ID":"7e30e07a-26a2-4051-9dba-6170acb64eba","Type":"ContainerDied","Data":"5f18a47765127888a6fb3cec4c9bbca017b4234700777938cb1199756c277bad"}
Sep 17 20:52:14.098837 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:14.098741    2575 scope.go:117] "RemoveContainer" containerID="5f18a47765127888a6fb3cec4c9bbca017b4234700777938cb1199756c277bad"
Sep 17 20:52:15.105781 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:15.105743    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-jknl5" event={"ID":"7e30e07a-26a2-4051-9dba-6170acb64eba","Type":"ContainerStarted","Data":"fb5963ea8dc4c4ecd585e61f35946342268cce696c6a0272912b44247e96fdf6"}
Sep 17 20:52:16.535236 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:16.535200    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-d7m9q_3afbe98f-5052-44e2-9d4b-da4bd2d30c3a/dns/0.log"
Sep 17 20:52:16.544080 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:16.544051    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:52:16.734031 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:16.734003    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-d7m9q_3afbe98f-5052-44e2-9d4b-da4bd2d30c3a/kube-rbac-proxy/0.log"
Sep 17 20:52:16.925000 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:16.924922    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-48p9f_5af35da6-3bf6-49ca-9a78-0218c7166c83/dns-node-resolver/0.log"
Sep 17 20:52:17.425354 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:17.425324    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:52:17.521678 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:17.521646    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:52:17.528422 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:17.528400    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8b75c8b68-8wx4k_7eb327ea-1cee-4481-8533-423762e782e0/router/0.log"
Sep 17 20:52:18.124967 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:18.124938    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-z4lgm_54ba4bcf-1779-4a9d-a5c3-d3bc4507466c/serve-healthcheck-canary/0.log"
Sep 17 20:52:18.327970 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:18.327942    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-685d7d68b5-55dn9_0c11c2c9-0efb-4522-89c3-6682fb58daa6/cluster-samples-operator/0.log"
Sep 17 20:52:18.476131 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:18.476102    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-67bbf8d54-844pn"
Sep 17 20:52:18.476284 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:18.476159    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-67bbf8d54-844pn"
Sep 17 20:52:18.526328 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:18.526299    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-685d7d68b5-55dn9_0c11c2c9-0efb-4522-89c3-6682fb58daa6/cluster-samples-operator-watch/0.log"
Sep 17 20:52:21.805316 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:21.805279    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:52:21.917780 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:21.917752    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:52:22.163338 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:22.163254    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:52:22.348202 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:22.348173    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:52:22.497554 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:22.497524    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:52:25.577776 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:25.577741    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:52:25.656167 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:25.656129    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:52:27.202714 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:27.202675    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:52:27.302934 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:27.302900    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:52:34.004276 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:34.004228    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:52:34.106610 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:34.106571    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:52:34.194066 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:34.194030    2575 generic.go:356] "Generic (PLEG): container finished" podID="08f37200-5c00-4946-8f1f-4b640e804e32" containerID="f411581fdab280b7f995255423d4ca967764c902545900674b75b79737413adc" exitCode=0
Sep 17 20:52:34.194141 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:34.194111    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-9mj28" event={"ID":"08f37200-5c00-4946-8f1f-4b640e804e32","Type":"ContainerDied","Data":"f411581fdab280b7f995255423d4ca967764c902545900674b75b79737413adc"}
Sep 17 20:52:34.194521 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:34.194507    2575 scope.go:117] "RemoveContainer" containerID="f411581fdab280b7f995255423d4ca967764c902545900674b75b79737413adc"
Sep 17 20:52:34.407770 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:34.407716    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6c4b45cb74-s4gzt" podUID="9dda82c0-e203-412d-8d70-bffaf6285281" containerName="registry" containerID="cri-o://51e9cd5e9a27a838c0ab939d252ced75e9aebe1a9813b8025bc87eadd82d508f" gracePeriod=30
Sep 17 20:52:34.649049 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:34.649026    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c4b45cb74-s4gzt"
Sep 17 20:52:34.759830 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:34.759797    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9dda82c0-e203-412d-8d70-bffaf6285281-image-registry-private-configuration\") pod \"9dda82c0-e203-412d-8d70-bffaf6285281\" (UID: \"9dda82c0-e203-412d-8d70-bffaf6285281\") "
Sep 17 20:52:34.759998 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:34.759893    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9dda82c0-e203-412d-8d70-bffaf6285281-registry-certificates\") pod \"9dda82c0-e203-412d-8d70-bffaf6285281\" (UID: \"9dda82c0-e203-412d-8d70-bffaf6285281\") "
Sep 17 20:52:34.759998 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:34.759931    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9dda82c0-e203-412d-8d70-bffaf6285281-ca-trust-extracted\") pod \"9dda82c0-e203-412d-8d70-bffaf6285281\" (UID: \"9dda82c0-e203-412d-8d70-bffaf6285281\") "
Sep 17 20:52:34.759998 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:34.759974    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bm9qs\" (UniqueName: \"kubernetes.io/projected/9dda82c0-e203-412d-8d70-bffaf6285281-kube-api-access-bm9qs\") pod \"9dda82c0-e203-412d-8d70-bffaf6285281\" (UID: \"9dda82c0-e203-412d-8d70-bffaf6285281\") "
Sep 17 20:52:34.760174 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:34.760000    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9dda82c0-e203-412d-8d70-bffaf6285281-installation-pull-secrets\") pod \"9dda82c0-e203-412d-8d70-bffaf6285281\" (UID: \"9dda82c0-e203-412d-8d70-bffaf6285281\") "
Sep 17 20:52:34.760174 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:34.760039    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9dda82c0-e203-412d-8d70-bffaf6285281-trusted-ca\") pod \"9dda82c0-e203-412d-8d70-bffaf6285281\" (UID: \"9dda82c0-e203-412d-8d70-bffaf6285281\") "
Sep 17 20:52:34.760174 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:34.760085    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9dda82c0-e203-412d-8d70-bffaf6285281-bound-sa-token\") pod \"9dda82c0-e203-412d-8d70-bffaf6285281\" (UID: \"9dda82c0-e203-412d-8d70-bffaf6285281\") "
Sep 17 20:52:34.760174 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:34.760113    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9dda82c0-e203-412d-8d70-bffaf6285281-registry-tls\") pod \"9dda82c0-e203-412d-8d70-bffaf6285281\" (UID: \"9dda82c0-e203-412d-8d70-bffaf6285281\") "
Sep 17 20:52:34.760589 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:34.760352    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dda82c0-e203-412d-8d70-bffaf6285281-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "9dda82c0-e203-412d-8d70-bffaf6285281" (UID: "9dda82c0-e203-412d-8d70-bffaf6285281"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:52:34.760589 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:34.760512    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dda82c0-e203-412d-8d70-bffaf6285281-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "9dda82c0-e203-412d-8d70-bffaf6285281" (UID: "9dda82c0-e203-412d-8d70-bffaf6285281"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:52:34.760797 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:34.760701    2575 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9dda82c0-e203-412d-8d70-bffaf6285281-registry-certificates\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 20:52:34.762640 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:34.762608    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dda82c0-e203-412d-8d70-bffaf6285281-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "9dda82c0-e203-412d-8d70-bffaf6285281" (UID: "9dda82c0-e203-412d-8d70-bffaf6285281"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:52:34.762735 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:34.762684    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dda82c0-e203-412d-8d70-bffaf6285281-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "9dda82c0-e203-412d-8d70-bffaf6285281" (UID: "9dda82c0-e203-412d-8d70-bffaf6285281"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:52:34.762805 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:34.762783    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dda82c0-e203-412d-8d70-bffaf6285281-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "9dda82c0-e203-412d-8d70-bffaf6285281" (UID: "9dda82c0-e203-412d-8d70-bffaf6285281"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:52:34.762805 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:34.762785    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dda82c0-e203-412d-8d70-bffaf6285281-kube-api-access-bm9qs" (OuterVolumeSpecName: "kube-api-access-bm9qs") pod "9dda82c0-e203-412d-8d70-bffaf6285281" (UID: "9dda82c0-e203-412d-8d70-bffaf6285281"). InnerVolumeSpecName "kube-api-access-bm9qs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:52:34.762928 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:34.762874    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dda82c0-e203-412d-8d70-bffaf6285281-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "9dda82c0-e203-412d-8d70-bffaf6285281" (UID: "9dda82c0-e203-412d-8d70-bffaf6285281"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:52:34.768368 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:34.768345    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dda82c0-e203-412d-8d70-bffaf6285281-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "9dda82c0-e203-412d-8d70-bffaf6285281" (UID: "9dda82c0-e203-412d-8d70-bffaf6285281"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:52:34.861347 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:34.861292    2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9dda82c0-e203-412d-8d70-bffaf6285281-trusted-ca\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 20:52:34.861347 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:34.861315    2575 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9dda82c0-e203-412d-8d70-bffaf6285281-bound-sa-token\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 20:52:34.861347 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:34.861323    2575 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9dda82c0-e203-412d-8d70-bffaf6285281-registry-tls\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 20:52:34.861347 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:34.861331    2575 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9dda82c0-e203-412d-8d70-bffaf6285281-image-registry-private-configuration\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 20:52:34.861347 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:34.861341    2575 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9dda82c0-e203-412d-8d70-bffaf6285281-ca-trust-extracted\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 20:52:34.861537 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:34.861350    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bm9qs\" (UniqueName: \"kubernetes.io/projected/9dda82c0-e203-412d-8d70-bffaf6285281-kube-api-access-bm9qs\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 20:52:34.861537 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:34.861359    2575 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9dda82c0-e203-412d-8d70-bffaf6285281-installation-pull-secrets\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 20:52:35.199494 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:35.199461    2575 generic.go:356] "Generic (PLEG): container finished" podID="9dda82c0-e203-412d-8d70-bffaf6285281" containerID="51e9cd5e9a27a838c0ab939d252ced75e9aebe1a9813b8025bc87eadd82d508f" exitCode=0
Sep 17 20:52:35.199942 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:35.199504    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c4b45cb74-s4gzt"
Sep 17 20:52:35.199942 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:35.199547    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c4b45cb74-s4gzt" event={"ID":"9dda82c0-e203-412d-8d70-bffaf6285281","Type":"ContainerDied","Data":"51e9cd5e9a27a838c0ab939d252ced75e9aebe1a9813b8025bc87eadd82d508f"}
Sep 17 20:52:35.199942 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:35.199600    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c4b45cb74-s4gzt" event={"ID":"9dda82c0-e203-412d-8d70-bffaf6285281","Type":"ContainerDied","Data":"5a02b7a1c53752968c30ae7225648eb460fe279946de56cd13fb476e0b3a174d"}
Sep 17 20:52:35.199942 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:35.199639    2575 scope.go:117] "RemoveContainer" containerID="51e9cd5e9a27a838c0ab939d252ced75e9aebe1a9813b8025bc87eadd82d508f"
Sep 17 20:52:35.201664 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:35.201635    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-9mj28" event={"ID":"08f37200-5c00-4946-8f1f-4b640e804e32","Type":"ContainerStarted","Data":"8f0d44b852dd4e7ec8faf2e2ffa9f5079fe116e77b3fbac4afd7265b69732250"}
Sep 17 20:52:35.203201 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:35.203180    2575 generic.go:356] "Generic (PLEG): container finished" podID="2d3961cf-90ef-46f5-b7e4-7af64d4d183e" containerID="4b79b51d3aac0fce1c544efe69b91ed060e8219a39d53a603f58dc0ad0ba46a3" exitCode=0
Sep 17 20:52:35.203328 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:35.203226    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-v4g7j" event={"ID":"2d3961cf-90ef-46f5-b7e4-7af64d4d183e","Type":"ContainerDied","Data":"4b79b51d3aac0fce1c544efe69b91ed060e8219a39d53a603f58dc0ad0ba46a3"}
Sep 17 20:52:35.203495 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:35.203480    2575 scope.go:117] "RemoveContainer" containerID="4b79b51d3aac0fce1c544efe69b91ed060e8219a39d53a603f58dc0ad0ba46a3"
Sep 17 20:52:35.209156 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:35.209136    2575 scope.go:117] "RemoveContainer" containerID="51e9cd5e9a27a838c0ab939d252ced75e9aebe1a9813b8025bc87eadd82d508f"
Sep 17 20:52:35.209445 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:52:35.209426    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51e9cd5e9a27a838c0ab939d252ced75e9aebe1a9813b8025bc87eadd82d508f\": container with ID starting with 51e9cd5e9a27a838c0ab939d252ced75e9aebe1a9813b8025bc87eadd82d508f not found: ID does not exist" containerID="51e9cd5e9a27a838c0ab939d252ced75e9aebe1a9813b8025bc87eadd82d508f"
Sep 17 20:52:35.209513 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:35.209456    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51e9cd5e9a27a838c0ab939d252ced75e9aebe1a9813b8025bc87eadd82d508f"} err="failed to get container status \"51e9cd5e9a27a838c0ab939d252ced75e9aebe1a9813b8025bc87eadd82d508f\": rpc error: code = NotFound desc = could not find container \"51e9cd5e9a27a838c0ab939d252ced75e9aebe1a9813b8025bc87eadd82d508f\": container with ID starting with 51e9cd5e9a27a838c0ab939d252ced75e9aebe1a9813b8025bc87eadd82d508f not found: ID does not exist"
Sep 17 20:52:35.326284 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:35.325930    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6c4b45cb74-s4gzt"]
Sep 17 20:52:35.344278 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:35.344249    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6c4b45cb74-s4gzt"]
Sep 17 20:52:35.355937 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:35.355911    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9dda82c0-e203-412d-8d70-bffaf6285281" path="/var/lib/kubelet/pods/9dda82c0-e203-412d-8d70-bffaf6285281/volumes"
Sep 17 20:52:36.209214 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:36.209180    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-v4g7j" event={"ID":"2d3961cf-90ef-46f5-b7e4-7af64d4d183e","Type":"ContainerStarted","Data":"487af021909b54f5e91e213f421e274e3cc3f17fad47259ed7afb65c36415823"}
Sep 17 20:52:38.481918 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:38.481891    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-67bbf8d54-844pn"
Sep 17 20:52:38.485647 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:38.485624    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-67bbf8d54-844pn"
Sep 17 20:52:40.548021 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:40.547988    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:52:40.612261 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:52:40.612230    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:53:07.109436 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:07.109395    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d42a790e-18b7-4478-b0c7-3c7cc4372fff-metrics-certs\") pod \"network-metrics-daemon-8b6dt\" (UID: \"d42a790e-18b7-4478-b0c7-3c7cc4372fff\") " pod="openshift-multus/network-metrics-daemon-8b6dt"
Sep 17 20:53:07.111669 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:07.111648    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d42a790e-18b7-4478-b0c7-3c7cc4372fff-metrics-certs\") pod \"network-metrics-daemon-8b6dt\" (UID: \"d42a790e-18b7-4478-b0c7-3c7cc4372fff\") " pod="openshift-multus/network-metrics-daemon-8b6dt"
Sep 17 20:53:07.254605 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:07.254574    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8j64n\""
Sep 17 20:53:07.263484 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:07.263462    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8b6dt"
Sep 17 20:53:07.534652 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:07.534624    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8b6dt"]
Sep 17 20:53:07.536742 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:53:07.536708    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd42a790e_18b7_4478_b0c7_3c7cc4372fff.slice/crio-a923c6bd478d1d5ff261dfedbcfe76fce5823b26a3e60daa97ec5365c43fcd91 WatchSource:0}: Error finding container a923c6bd478d1d5ff261dfedbcfe76fce5823b26a3e60daa97ec5365c43fcd91: Status 404 returned error can't find the container with id a923c6bd478d1d5ff261dfedbcfe76fce5823b26a3e60daa97ec5365c43fcd91
Sep 17 20:53:08.335384 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:08.335337    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8b6dt" event={"ID":"d42a790e-18b7-4478-b0c7-3c7cc4372fff","Type":"ContainerStarted","Data":"a923c6bd478d1d5ff261dfedbcfe76fce5823b26a3e60daa97ec5365c43fcd91"}
Sep 17 20:53:09.341265 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:09.341231    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8b6dt" event={"ID":"d42a790e-18b7-4478-b0c7-3c7cc4372fff","Type":"ContainerStarted","Data":"40f021a4b6a18dbea23866ec1ef1c9928581e76aea02e0038399932bc37b8f0f"}
Sep 17 20:53:09.341626 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:09.341287    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8b6dt" event={"ID":"d42a790e-18b7-4478-b0c7-3c7cc4372fff","Type":"ContainerStarted","Data":"61e3eb3435549d8acd98615d8fc46e43bcd5d840bd29c51e18768986ccda9c31"}
Sep 17 20:53:09.362350 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:09.362306    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-8b6dt" podStartSLOduration=253.371170499 podStartE2EDuration="4m14.362291307s" totalImagesPullingTime="991.120808ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:48:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:53:09.361194518 +0000 UTC m=+254.513638896" watchObservedRunningTime="2026-09-17 20:53:09.362291307 +0000 UTC m=+254.514735683"
Sep 17 20:53:09.466580 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:09.466545    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:53:09.535338 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:09.535307    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:53:13.009862 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:13.009813    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:53:13.353764 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:13.353644    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:53:14.082468 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:14.082429    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 20:53:14.083030 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:14.082743    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="49b464f9-52b0-49b6-a3ff-a748723f65ee" containerName="alertmanager" containerID="cri-o://1760014afb019f8d1d6cd742ad4b68d3edfd366e776ad96b2e994eb0dd4cdbd2" gracePeriod=120
Sep 17 20:53:14.083030 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:14.082809    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="49b464f9-52b0-49b6-a3ff-a748723f65ee" containerName="kube-rbac-proxy" containerID="cri-o://8fe241f7e25e7e1bc5ce3b9f0c78d0d230f0f24637ed3b915c3862dfa1d2284e" gracePeriod=120
Sep 17 20:53:14.083030 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:14.082837    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="49b464f9-52b0-49b6-a3ff-a748723f65ee" containerName="config-reloader" containerID="cri-o://c373f74f20d20157e5369f7f2486198d1e026aa58b3d8b4067e63db567307cf5" gracePeriod=120
Sep 17 20:53:14.083030 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:14.082815    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="49b464f9-52b0-49b6-a3ff-a748723f65ee" containerName="kube-rbac-proxy-metric" containerID="cri-o://84e65b0000c7155632fb0eee5568c1a583c876fb2b24c6560e6bf2286566398d" gracePeriod=120
Sep 17 20:53:14.083030 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:14.082888    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="49b464f9-52b0-49b6-a3ff-a748723f65ee" containerName="prom-label-proxy" containerID="cri-o://f50c083b13d615a658b051ec57092cc96da60085b4224b8f2250fb270406a2b9" gracePeriod=120
Sep 17 20:53:14.083030 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:14.082815    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="49b464f9-52b0-49b6-a3ff-a748723f65ee" containerName="kube-rbac-proxy-web" containerID="cri-o://0b39670623a99267e9e72bfc5d3dd36dc89d6446518f71e949d855450624b303" gracePeriod=120
Sep 17 20:53:14.364634 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:14.364554    2575 generic.go:356] "Generic (PLEG): container finished" podID="49b464f9-52b0-49b6-a3ff-a748723f65ee" containerID="f50c083b13d615a658b051ec57092cc96da60085b4224b8f2250fb270406a2b9" exitCode=0
Sep 17 20:53:14.364634 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:14.364578    2575 generic.go:356] "Generic (PLEG): container finished" podID="49b464f9-52b0-49b6-a3ff-a748723f65ee" containerID="8fe241f7e25e7e1bc5ce3b9f0c78d0d230f0f24637ed3b915c3862dfa1d2284e" exitCode=0
Sep 17 20:53:14.364634 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:14.364584    2575 generic.go:356] "Generic (PLEG): container finished" podID="49b464f9-52b0-49b6-a3ff-a748723f65ee" containerID="c373f74f20d20157e5369f7f2486198d1e026aa58b3d8b4067e63db567307cf5" exitCode=0
Sep 17 20:53:14.364634 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:14.364590    2575 generic.go:356] "Generic (PLEG): container finished" podID="49b464f9-52b0-49b6-a3ff-a748723f65ee" containerID="1760014afb019f8d1d6cd742ad4b68d3edfd366e776ad96b2e994eb0dd4cdbd2" exitCode=0
Sep 17 20:53:14.364634 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:14.364625    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"49b464f9-52b0-49b6-a3ff-a748723f65ee","Type":"ContainerDied","Data":"f50c083b13d615a658b051ec57092cc96da60085b4224b8f2250fb270406a2b9"}
Sep 17 20:53:14.364916 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:14.364683    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"49b464f9-52b0-49b6-a3ff-a748723f65ee","Type":"ContainerDied","Data":"8fe241f7e25e7e1bc5ce3b9f0c78d0d230f0f24637ed3b915c3862dfa1d2284e"}
Sep 17 20:53:14.364916 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:14.364709    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"49b464f9-52b0-49b6-a3ff-a748723f65ee","Type":"ContainerDied","Data":"c373f74f20d20157e5369f7f2486198d1e026aa58b3d8b4067e63db567307cf5"}
Sep 17 20:53:14.364916 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:14.364733    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"49b464f9-52b0-49b6-a3ff-a748723f65ee","Type":"ContainerDied","Data":"1760014afb019f8d1d6cd742ad4b68d3edfd366e776ad96b2e994eb0dd4cdbd2"}
Sep 17 20:53:15.326591 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.326565    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:15.373805 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.373686    2575 generic.go:356] "Generic (PLEG): container finished" podID="49b464f9-52b0-49b6-a3ff-a748723f65ee" containerID="84e65b0000c7155632fb0eee5568c1a583c876fb2b24c6560e6bf2286566398d" exitCode=0
Sep 17 20:53:15.373805 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.373716    2575 generic.go:356] "Generic (PLEG): container finished" podID="49b464f9-52b0-49b6-a3ff-a748723f65ee" containerID="0b39670623a99267e9e72bfc5d3dd36dc89d6446518f71e949d855450624b303" exitCode=0
Sep 17 20:53:15.373805 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.373716    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"49b464f9-52b0-49b6-a3ff-a748723f65ee","Type":"ContainerDied","Data":"84e65b0000c7155632fb0eee5568c1a583c876fb2b24c6560e6bf2286566398d"}
Sep 17 20:53:15.373805 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.373792    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"49b464f9-52b0-49b6-a3ff-a748723f65ee","Type":"ContainerDied","Data":"0b39670623a99267e9e72bfc5d3dd36dc89d6446518f71e949d855450624b303"}
Sep 17 20:53:15.373805 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.373804    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:15.374205 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.373827    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"49b464f9-52b0-49b6-a3ff-a748723f65ee","Type":"ContainerDied","Data":"090d24a84103d31c54b571844e81ecd8ef7760aa607ce43e9d4240517b073d0c"}
Sep 17 20:53:15.374205 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.373884    2575 scope.go:117] "RemoveContainer" containerID="f50c083b13d615a658b051ec57092cc96da60085b4224b8f2250fb270406a2b9"
Sep 17 20:53:15.389792 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.389769    2575 scope.go:117] "RemoveContainer" containerID="84e65b0000c7155632fb0eee5568c1a583c876fb2b24c6560e6bf2286566398d"
Sep 17 20:53:15.398166 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.398147    2575 scope.go:117] "RemoveContainer" containerID="8fe241f7e25e7e1bc5ce3b9f0c78d0d230f0f24637ed3b915c3862dfa1d2284e"
Sep 17 20:53:15.405024 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.405005    2575 scope.go:117] "RemoveContainer" containerID="0b39670623a99267e9e72bfc5d3dd36dc89d6446518f71e949d855450624b303"
Sep 17 20:53:15.411730 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.411711    2575 scope.go:117] "RemoveContainer" containerID="c373f74f20d20157e5369f7f2486198d1e026aa58b3d8b4067e63db567307cf5"
Sep 17 20:53:15.418638 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.418621    2575 scope.go:117] "RemoveContainer" containerID="1760014afb019f8d1d6cd742ad4b68d3edfd366e776ad96b2e994eb0dd4cdbd2"
Sep 17 20:53:15.426379 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.426355    2575 scope.go:117] "RemoveContainer" containerID="5f8f794442979f1ad6b90d6cc897660c0db739c62c86776ae476d8d1ac807f45"
Sep 17 20:53:15.433394 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.433367    2575 scope.go:117] "RemoveContainer" containerID="f50c083b13d615a658b051ec57092cc96da60085b4224b8f2250fb270406a2b9"
Sep 17 20:53:15.433662 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:53:15.433644    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f50c083b13d615a658b051ec57092cc96da60085b4224b8f2250fb270406a2b9\": container with ID starting with f50c083b13d615a658b051ec57092cc96da60085b4224b8f2250fb270406a2b9 not found: ID does not exist" containerID="f50c083b13d615a658b051ec57092cc96da60085b4224b8f2250fb270406a2b9"
Sep 17 20:53:15.433716 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.433673    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f50c083b13d615a658b051ec57092cc96da60085b4224b8f2250fb270406a2b9"} err="failed to get container status \"f50c083b13d615a658b051ec57092cc96da60085b4224b8f2250fb270406a2b9\": rpc error: code = NotFound desc = could not find container \"f50c083b13d615a658b051ec57092cc96da60085b4224b8f2250fb270406a2b9\": container with ID starting with f50c083b13d615a658b051ec57092cc96da60085b4224b8f2250fb270406a2b9 not found: ID does not exist"
Sep 17 20:53:15.433716 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.433690    2575 scope.go:117] "RemoveContainer" containerID="84e65b0000c7155632fb0eee5568c1a583c876fb2b24c6560e6bf2286566398d"
Sep 17 20:53:15.433993 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:53:15.433968    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84e65b0000c7155632fb0eee5568c1a583c876fb2b24c6560e6bf2286566398d\": container with ID starting with 84e65b0000c7155632fb0eee5568c1a583c876fb2b24c6560e6bf2286566398d not found: ID does not exist" containerID="84e65b0000c7155632fb0eee5568c1a583c876fb2b24c6560e6bf2286566398d"
Sep 17 20:53:15.434068 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.434004    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84e65b0000c7155632fb0eee5568c1a583c876fb2b24c6560e6bf2286566398d"} err="failed to get container status \"84e65b0000c7155632fb0eee5568c1a583c876fb2b24c6560e6bf2286566398d\": rpc error: code = NotFound desc = could not find container \"84e65b0000c7155632fb0eee5568c1a583c876fb2b24c6560e6bf2286566398d\": container with ID starting with 84e65b0000c7155632fb0eee5568c1a583c876fb2b24c6560e6bf2286566398d not found: ID does not exist"
Sep 17 20:53:15.434068 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.434023    2575 scope.go:117] "RemoveContainer" containerID="8fe241f7e25e7e1bc5ce3b9f0c78d0d230f0f24637ed3b915c3862dfa1d2284e"
Sep 17 20:53:15.434288 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:53:15.434270    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fe241f7e25e7e1bc5ce3b9f0c78d0d230f0f24637ed3b915c3862dfa1d2284e\": container with ID starting with 8fe241f7e25e7e1bc5ce3b9f0c78d0d230f0f24637ed3b915c3862dfa1d2284e not found: ID does not exist" containerID="8fe241f7e25e7e1bc5ce3b9f0c78d0d230f0f24637ed3b915c3862dfa1d2284e"
Sep 17 20:53:15.434341 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.434293    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fe241f7e25e7e1bc5ce3b9f0c78d0d230f0f24637ed3b915c3862dfa1d2284e"} err="failed to get container status \"8fe241f7e25e7e1bc5ce3b9f0c78d0d230f0f24637ed3b915c3862dfa1d2284e\": rpc error: code = NotFound desc = could not find container \"8fe241f7e25e7e1bc5ce3b9f0c78d0d230f0f24637ed3b915c3862dfa1d2284e\": container with ID starting with 8fe241f7e25e7e1bc5ce3b9f0c78d0d230f0f24637ed3b915c3862dfa1d2284e not found: ID does not exist"
Sep 17 20:53:15.434341 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.434308    2575 scope.go:117] "RemoveContainer" containerID="0b39670623a99267e9e72bfc5d3dd36dc89d6446518f71e949d855450624b303"
Sep 17 20:53:15.434545 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:53:15.434530    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b39670623a99267e9e72bfc5d3dd36dc89d6446518f71e949d855450624b303\": container with ID starting with 0b39670623a99267e9e72bfc5d3dd36dc89d6446518f71e949d855450624b303 not found: ID does not exist" containerID="0b39670623a99267e9e72bfc5d3dd36dc89d6446518f71e949d855450624b303"
Sep 17 20:53:15.434598 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.434548    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b39670623a99267e9e72bfc5d3dd36dc89d6446518f71e949d855450624b303"} err="failed to get container status \"0b39670623a99267e9e72bfc5d3dd36dc89d6446518f71e949d855450624b303\": rpc error: code = NotFound desc = could not find container \"0b39670623a99267e9e72bfc5d3dd36dc89d6446518f71e949d855450624b303\": container with ID starting with 0b39670623a99267e9e72bfc5d3dd36dc89d6446518f71e949d855450624b303 not found: ID does not exist"
Sep 17 20:53:15.434598 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.434562    2575 scope.go:117] "RemoveContainer" containerID="c373f74f20d20157e5369f7f2486198d1e026aa58b3d8b4067e63db567307cf5"
Sep 17 20:53:15.434820 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:53:15.434805    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c373f74f20d20157e5369f7f2486198d1e026aa58b3d8b4067e63db567307cf5\": container with ID starting with c373f74f20d20157e5369f7f2486198d1e026aa58b3d8b4067e63db567307cf5 not found: ID does not exist" containerID="c373f74f20d20157e5369f7f2486198d1e026aa58b3d8b4067e63db567307cf5"
Sep 17 20:53:15.434909 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.434822    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c373f74f20d20157e5369f7f2486198d1e026aa58b3d8b4067e63db567307cf5"} err="failed to get container status \"c373f74f20d20157e5369f7f2486198d1e026aa58b3d8b4067e63db567307cf5\": rpc error: code = NotFound desc = could not find container \"c373f74f20d20157e5369f7f2486198d1e026aa58b3d8b4067e63db567307cf5\": container with ID starting with c373f74f20d20157e5369f7f2486198d1e026aa58b3d8b4067e63db567307cf5 not found: ID does not exist"
Sep 17 20:53:15.434909 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.434843    2575 scope.go:117] "RemoveContainer" containerID="1760014afb019f8d1d6cd742ad4b68d3edfd366e776ad96b2e994eb0dd4cdbd2"
Sep 17 20:53:15.435102 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:53:15.435086    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1760014afb019f8d1d6cd742ad4b68d3edfd366e776ad96b2e994eb0dd4cdbd2\": container with ID starting with 1760014afb019f8d1d6cd742ad4b68d3edfd366e776ad96b2e994eb0dd4cdbd2 not found: ID does not exist" containerID="1760014afb019f8d1d6cd742ad4b68d3edfd366e776ad96b2e994eb0dd4cdbd2"
Sep 17 20:53:15.435142 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.435106    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1760014afb019f8d1d6cd742ad4b68d3edfd366e776ad96b2e994eb0dd4cdbd2"} err="failed to get container status \"1760014afb019f8d1d6cd742ad4b68d3edfd366e776ad96b2e994eb0dd4cdbd2\": rpc error: code = NotFound desc = could not find container \"1760014afb019f8d1d6cd742ad4b68d3edfd366e776ad96b2e994eb0dd4cdbd2\": container with ID starting with 1760014afb019f8d1d6cd742ad4b68d3edfd366e776ad96b2e994eb0dd4cdbd2 not found: ID does not exist"
Sep 17 20:53:15.435142 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.435120    2575 scope.go:117] "RemoveContainer" containerID="5f8f794442979f1ad6b90d6cc897660c0db739c62c86776ae476d8d1ac807f45"
Sep 17 20:53:15.435396 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:53:15.435375    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f8f794442979f1ad6b90d6cc897660c0db739c62c86776ae476d8d1ac807f45\": container with ID starting with 5f8f794442979f1ad6b90d6cc897660c0db739c62c86776ae476d8d1ac807f45 not found: ID does not exist" containerID="5f8f794442979f1ad6b90d6cc897660c0db739c62c86776ae476d8d1ac807f45"
Sep 17 20:53:15.435464 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.435404    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f8f794442979f1ad6b90d6cc897660c0db739c62c86776ae476d8d1ac807f45"} err="failed to get container status \"5f8f794442979f1ad6b90d6cc897660c0db739c62c86776ae476d8d1ac807f45\": rpc error: code = NotFound desc = could not find container \"5f8f794442979f1ad6b90d6cc897660c0db739c62c86776ae476d8d1ac807f45\": container with ID starting with 5f8f794442979f1ad6b90d6cc897660c0db739c62c86776ae476d8d1ac807f45 not found: ID does not exist"
Sep 17 20:53:15.435464 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.435425    2575 scope.go:117] "RemoveContainer" containerID="f50c083b13d615a658b051ec57092cc96da60085b4224b8f2250fb270406a2b9"
Sep 17 20:53:15.435695 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.435678    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f50c083b13d615a658b051ec57092cc96da60085b4224b8f2250fb270406a2b9"} err="failed to get container status \"f50c083b13d615a658b051ec57092cc96da60085b4224b8f2250fb270406a2b9\": rpc error: code = NotFound desc = could not find container \"f50c083b13d615a658b051ec57092cc96da60085b4224b8f2250fb270406a2b9\": container with ID starting with f50c083b13d615a658b051ec57092cc96da60085b4224b8f2250fb270406a2b9 not found: ID does not exist"
Sep 17 20:53:15.435746 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.435696    2575 scope.go:117] "RemoveContainer" containerID="84e65b0000c7155632fb0eee5568c1a583c876fb2b24c6560e6bf2286566398d"
Sep 17 20:53:15.436007 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.435990    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84e65b0000c7155632fb0eee5568c1a583c876fb2b24c6560e6bf2286566398d"} err="failed to get container status \"84e65b0000c7155632fb0eee5568c1a583c876fb2b24c6560e6bf2286566398d\": rpc error: code = NotFound desc = could not find container \"84e65b0000c7155632fb0eee5568c1a583c876fb2b24c6560e6bf2286566398d\": container with ID starting with 84e65b0000c7155632fb0eee5568c1a583c876fb2b24c6560e6bf2286566398d not found: ID does not exist"
Sep 17 20:53:15.436066 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.436006    2575 scope.go:117] "RemoveContainer" containerID="8fe241f7e25e7e1bc5ce3b9f0c78d0d230f0f24637ed3b915c3862dfa1d2284e"
Sep 17 20:53:15.436271 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.436252    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fe241f7e25e7e1bc5ce3b9f0c78d0d230f0f24637ed3b915c3862dfa1d2284e"} err="failed to get container status \"8fe241f7e25e7e1bc5ce3b9f0c78d0d230f0f24637ed3b915c3862dfa1d2284e\": rpc error: code = NotFound desc = could not find container \"8fe241f7e25e7e1bc5ce3b9f0c78d0d230f0f24637ed3b915c3862dfa1d2284e\": container with ID starting with 8fe241f7e25e7e1bc5ce3b9f0c78d0d230f0f24637ed3b915c3862dfa1d2284e not found: ID does not exist"
Sep 17 20:53:15.436355 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.436273    2575 scope.go:117] "RemoveContainer" containerID="0b39670623a99267e9e72bfc5d3dd36dc89d6446518f71e949d855450624b303"
Sep 17 20:53:15.436511 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.436492    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b39670623a99267e9e72bfc5d3dd36dc89d6446518f71e949d855450624b303"} err="failed to get container status \"0b39670623a99267e9e72bfc5d3dd36dc89d6446518f71e949d855450624b303\": rpc error: code = NotFound desc = could not find container \"0b39670623a99267e9e72bfc5d3dd36dc89d6446518f71e949d855450624b303\": container with ID starting with 0b39670623a99267e9e72bfc5d3dd36dc89d6446518f71e949d855450624b303 not found: ID does not exist"
Sep 17 20:53:15.436511 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.436508    2575 scope.go:117] "RemoveContainer" containerID="c373f74f20d20157e5369f7f2486198d1e026aa58b3d8b4067e63db567307cf5"
Sep 17 20:53:15.436720 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.436705    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c373f74f20d20157e5369f7f2486198d1e026aa58b3d8b4067e63db567307cf5"} err="failed to get container status \"c373f74f20d20157e5369f7f2486198d1e026aa58b3d8b4067e63db567307cf5\": rpc error: code = NotFound desc = could not find container \"c373f74f20d20157e5369f7f2486198d1e026aa58b3d8b4067e63db567307cf5\": container with ID starting with c373f74f20d20157e5369f7f2486198d1e026aa58b3d8b4067e63db567307cf5 not found: ID does not exist"
Sep 17 20:53:15.436720 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.436720    2575 scope.go:117] "RemoveContainer" containerID="1760014afb019f8d1d6cd742ad4b68d3edfd366e776ad96b2e994eb0dd4cdbd2"
Sep 17 20:53:15.436969 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.436954    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1760014afb019f8d1d6cd742ad4b68d3edfd366e776ad96b2e994eb0dd4cdbd2"} err="failed to get container status \"1760014afb019f8d1d6cd742ad4b68d3edfd366e776ad96b2e994eb0dd4cdbd2\": rpc error: code = NotFound desc = could not find container \"1760014afb019f8d1d6cd742ad4b68d3edfd366e776ad96b2e994eb0dd4cdbd2\": container with ID starting with 1760014afb019f8d1d6cd742ad4b68d3edfd366e776ad96b2e994eb0dd4cdbd2 not found: ID does not exist"
Sep 17 20:53:15.437020 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.436969    2575 scope.go:117] "RemoveContainer" containerID="5f8f794442979f1ad6b90d6cc897660c0db739c62c86776ae476d8d1ac807f45"
Sep 17 20:53:15.437179 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.437161    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f8f794442979f1ad6b90d6cc897660c0db739c62c86776ae476d8d1ac807f45"} err="failed to get container status \"5f8f794442979f1ad6b90d6cc897660c0db739c62c86776ae476d8d1ac807f45\": rpc error: code = NotFound desc = could not find container \"5f8f794442979f1ad6b90d6cc897660c0db739c62c86776ae476d8d1ac807f45\": container with ID starting with 5f8f794442979f1ad6b90d6cc897660c0db739c62c86776ae476d8d1ac807f45 not found: ID does not exist"
Sep 17 20:53:15.480743 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.480715    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/49b464f9-52b0-49b6-a3ff-a748723f65ee-secret-alertmanager-kube-rbac-proxy-metric\") pod \"49b464f9-52b0-49b6-a3ff-a748723f65ee\" (UID: \"49b464f9-52b0-49b6-a3ff-a748723f65ee\") "
Sep 17 20:53:15.480906 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.480752    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/49b464f9-52b0-49b6-a3ff-a748723f65ee-secret-alertmanager-main-tls\") pod \"49b464f9-52b0-49b6-a3ff-a748723f65ee\" (UID: \"49b464f9-52b0-49b6-a3ff-a748723f65ee\") "
Sep 17 20:53:15.480906 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.480800    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/49b464f9-52b0-49b6-a3ff-a748723f65ee-secret-alertmanager-kube-rbac-proxy-web\") pod \"49b464f9-52b0-49b6-a3ff-a748723f65ee\" (UID: \"49b464f9-52b0-49b6-a3ff-a748723f65ee\") "
Sep 17 20:53:15.480906 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.480834    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/49b464f9-52b0-49b6-a3ff-a748723f65ee-cluster-tls-config\") pod \"49b464f9-52b0-49b6-a3ff-a748723f65ee\" (UID: \"49b464f9-52b0-49b6-a3ff-a748723f65ee\") "
Sep 17 20:53:15.480906 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.480884    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/49b464f9-52b0-49b6-a3ff-a748723f65ee-web-config\") pod \"49b464f9-52b0-49b6-a3ff-a748723f65ee\" (UID: \"49b464f9-52b0-49b6-a3ff-a748723f65ee\") "
Sep 17 20:53:15.481117 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.480918    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p86bm\" (UniqueName: \"kubernetes.io/projected/49b464f9-52b0-49b6-a3ff-a748723f65ee-kube-api-access-p86bm\") pod \"49b464f9-52b0-49b6-a3ff-a748723f65ee\" (UID: \"49b464f9-52b0-49b6-a3ff-a748723f65ee\") "
Sep 17 20:53:15.481117 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.480948    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49b464f9-52b0-49b6-a3ff-a748723f65ee-alertmanager-trusted-ca-bundle\") pod \"49b464f9-52b0-49b6-a3ff-a748723f65ee\" (UID: \"49b464f9-52b0-49b6-a3ff-a748723f65ee\") "
Sep 17 20:53:15.481117 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.481007    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/49b464f9-52b0-49b6-a3ff-a748723f65ee-config-out\") pod \"49b464f9-52b0-49b6-a3ff-a748723f65ee\" (UID: \"49b464f9-52b0-49b6-a3ff-a748723f65ee\") "
Sep 17 20:53:15.481117 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.481033    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/49b464f9-52b0-49b6-a3ff-a748723f65ee-config-volume\") pod \"49b464f9-52b0-49b6-a3ff-a748723f65ee\" (UID: \"49b464f9-52b0-49b6-a3ff-a748723f65ee\") "
Sep 17 20:53:15.481117 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.481071    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/49b464f9-52b0-49b6-a3ff-a748723f65ee-metrics-client-ca\") pod \"49b464f9-52b0-49b6-a3ff-a748723f65ee\" (UID: \"49b464f9-52b0-49b6-a3ff-a748723f65ee\") "
Sep 17 20:53:15.481117 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.481097    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/49b464f9-52b0-49b6-a3ff-a748723f65ee-secret-alertmanager-kube-rbac-proxy\") pod \"49b464f9-52b0-49b6-a3ff-a748723f65ee\" (UID: \"49b464f9-52b0-49b6-a3ff-a748723f65ee\") "
Sep 17 20:53:15.481393 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.481127    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/49b464f9-52b0-49b6-a3ff-a748723f65ee-tls-assets\") pod \"49b464f9-52b0-49b6-a3ff-a748723f65ee\" (UID: \"49b464f9-52b0-49b6-a3ff-a748723f65ee\") "
Sep 17 20:53:15.481393 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.481156    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/49b464f9-52b0-49b6-a3ff-a748723f65ee-alertmanager-main-db\") pod \"49b464f9-52b0-49b6-a3ff-a748723f65ee\" (UID: \"49b464f9-52b0-49b6-a3ff-a748723f65ee\") "
Sep 17 20:53:15.481393 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.481369    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49b464f9-52b0-49b6-a3ff-a748723f65ee-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "49b464f9-52b0-49b6-a3ff-a748723f65ee" (UID: "49b464f9-52b0-49b6-a3ff-a748723f65ee"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:53:15.481537 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.481471    2575 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49b464f9-52b0-49b6-a3ff-a748723f65ee-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 20:53:15.482492 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.482163    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49b464f9-52b0-49b6-a3ff-a748723f65ee-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "49b464f9-52b0-49b6-a3ff-a748723f65ee" (UID: "49b464f9-52b0-49b6-a3ff-a748723f65ee"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:53:15.482615 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.482542    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49b464f9-52b0-49b6-a3ff-a748723f65ee-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "49b464f9-52b0-49b6-a3ff-a748723f65ee" (UID: "49b464f9-52b0-49b6-a3ff-a748723f65ee"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:53:15.484164 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.484135    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49b464f9-52b0-49b6-a3ff-a748723f65ee-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "49b464f9-52b0-49b6-a3ff-a748723f65ee" (UID: "49b464f9-52b0-49b6-a3ff-a748723f65ee"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:53:15.484314 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.484291    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49b464f9-52b0-49b6-a3ff-a748723f65ee-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "49b464f9-52b0-49b6-a3ff-a748723f65ee" (UID: "49b464f9-52b0-49b6-a3ff-a748723f65ee"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:53:15.484432 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.484290    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49b464f9-52b0-49b6-a3ff-a748723f65ee-kube-api-access-p86bm" (OuterVolumeSpecName: "kube-api-access-p86bm") pod "49b464f9-52b0-49b6-a3ff-a748723f65ee" (UID: "49b464f9-52b0-49b6-a3ff-a748723f65ee"). InnerVolumeSpecName "kube-api-access-p86bm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:53:15.484672 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.484648    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49b464f9-52b0-49b6-a3ff-a748723f65ee-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "49b464f9-52b0-49b6-a3ff-a748723f65ee" (UID: "49b464f9-52b0-49b6-a3ff-a748723f65ee"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:53:15.485251 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.485224    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49b464f9-52b0-49b6-a3ff-a748723f65ee-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "49b464f9-52b0-49b6-a3ff-a748723f65ee" (UID: "49b464f9-52b0-49b6-a3ff-a748723f65ee"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:53:15.485394 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.485361    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49b464f9-52b0-49b6-a3ff-a748723f65ee-config-out" (OuterVolumeSpecName: "config-out") pod "49b464f9-52b0-49b6-a3ff-a748723f65ee" (UID: "49b464f9-52b0-49b6-a3ff-a748723f65ee"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:53:15.485677 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.485658    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49b464f9-52b0-49b6-a3ff-a748723f65ee-config-volume" (OuterVolumeSpecName: "config-volume") pod "49b464f9-52b0-49b6-a3ff-a748723f65ee" (UID: "49b464f9-52b0-49b6-a3ff-a748723f65ee"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:53:15.485889 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.485870    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49b464f9-52b0-49b6-a3ff-a748723f65ee-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "49b464f9-52b0-49b6-a3ff-a748723f65ee" (UID: "49b464f9-52b0-49b6-a3ff-a748723f65ee"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:53:15.488501 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.488480    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49b464f9-52b0-49b6-a3ff-a748723f65ee-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "49b464f9-52b0-49b6-a3ff-a748723f65ee" (UID: "49b464f9-52b0-49b6-a3ff-a748723f65ee"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:53:15.495086 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.495061    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49b464f9-52b0-49b6-a3ff-a748723f65ee-web-config" (OuterVolumeSpecName: "web-config") pod "49b464f9-52b0-49b6-a3ff-a748723f65ee" (UID: "49b464f9-52b0-49b6-a3ff-a748723f65ee"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:53:15.582813 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.582776    2575 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/49b464f9-52b0-49b6-a3ff-a748723f65ee-cluster-tls-config\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 20:53:15.582813 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.582806    2575 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/49b464f9-52b0-49b6-a3ff-a748723f65ee-web-config\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 20:53:15.582813 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.582816    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p86bm\" (UniqueName: \"kubernetes.io/projected/49b464f9-52b0-49b6-a3ff-a748723f65ee-kube-api-access-p86bm\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 20:53:15.583047 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.582825    2575 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/49b464f9-52b0-49b6-a3ff-a748723f65ee-config-out\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 20:53:15.583047 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.582833    2575 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/49b464f9-52b0-49b6-a3ff-a748723f65ee-config-volume\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 20:53:15.583047 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.582841    2575 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/49b464f9-52b0-49b6-a3ff-a748723f65ee-metrics-client-ca\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 20:53:15.583047 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.582867    2575 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/49b464f9-52b0-49b6-a3ff-a748723f65ee-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 20:53:15.583047 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.582877    2575 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/49b464f9-52b0-49b6-a3ff-a748723f65ee-tls-assets\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 20:53:15.583047 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.582885    2575 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/49b464f9-52b0-49b6-a3ff-a748723f65ee-alertmanager-main-db\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 20:53:15.583047 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.582893    2575 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/49b464f9-52b0-49b6-a3ff-a748723f65ee-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 20:53:15.583047 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.582902    2575 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/49b464f9-52b0-49b6-a3ff-a748723f65ee-secret-alertmanager-main-tls\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 20:53:15.583047 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.582910    2575 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/49b464f9-52b0-49b6-a3ff-a748723f65ee-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 20:53:15.703948 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.703908    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 20:53:15.706001 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.705978    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 20:53:15.732734 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.732708    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 20:53:15.733018 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.733006    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49b464f9-52b0-49b6-a3ff-a748723f65ee" containerName="alertmanager"
Sep 17 20:53:15.733059 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.733020    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="49b464f9-52b0-49b6-a3ff-a748723f65ee" containerName="alertmanager"
Sep 17 20:53:15.733059 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.733037    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49b464f9-52b0-49b6-a3ff-a748723f65ee" containerName="kube-rbac-proxy"
Sep 17 20:53:15.733059 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.733044    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="49b464f9-52b0-49b6-a3ff-a748723f65ee" containerName="kube-rbac-proxy"
Sep 17 20:53:15.733059 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.733051    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49b464f9-52b0-49b6-a3ff-a748723f65ee" containerName="kube-rbac-proxy-web"
Sep 17 20:53:15.733059 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.733057    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="49b464f9-52b0-49b6-a3ff-a748723f65ee" containerName="kube-rbac-proxy-web"
Sep 17 20:53:15.733209 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.733064    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49b464f9-52b0-49b6-a3ff-a748723f65ee" containerName="init-config-reloader"
Sep 17 20:53:15.733209 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.733070    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="49b464f9-52b0-49b6-a3ff-a748723f65ee" containerName="init-config-reloader"
Sep 17 20:53:15.733209 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.733077    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49b464f9-52b0-49b6-a3ff-a748723f65ee" containerName="prom-label-proxy"
Sep 17 20:53:15.733209 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.733082    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="49b464f9-52b0-49b6-a3ff-a748723f65ee" containerName="prom-label-proxy"
Sep 17 20:53:15.733209 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.733091    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9dda82c0-e203-412d-8d70-bffaf6285281" containerName="registry"
Sep 17 20:53:15.733209 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.733096    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9dda82c0-e203-412d-8d70-bffaf6285281" containerName="registry"
Sep 17 20:53:15.733209 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.733101    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49b464f9-52b0-49b6-a3ff-a748723f65ee" containerName="config-reloader"
Sep 17 20:53:15.733209 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.733106    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="49b464f9-52b0-49b6-a3ff-a748723f65ee" containerName="config-reloader"
Sep 17 20:53:15.733209 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.733114    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49b464f9-52b0-49b6-a3ff-a748723f65ee" containerName="kube-rbac-proxy-metric"
Sep 17 20:53:15.733209 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.733119    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="49b464f9-52b0-49b6-a3ff-a748723f65ee" containerName="kube-rbac-proxy-metric"
Sep 17 20:53:15.733209 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.733164    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="9dda82c0-e203-412d-8d70-bffaf6285281" containerName="registry"
Sep 17 20:53:15.733209 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.733171    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="49b464f9-52b0-49b6-a3ff-a748723f65ee" containerName="alertmanager"
Sep 17 20:53:15.733209 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.733177    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="49b464f9-52b0-49b6-a3ff-a748723f65ee" containerName="kube-rbac-proxy"
Sep 17 20:53:15.733209 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.733184    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="49b464f9-52b0-49b6-a3ff-a748723f65ee" containerName="kube-rbac-proxy-web"
Sep 17 20:53:15.733209 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.733189    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="49b464f9-52b0-49b6-a3ff-a748723f65ee" containerName="kube-rbac-proxy-metric"
Sep 17 20:53:15.733209 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.733194    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="49b464f9-52b0-49b6-a3ff-a748723f65ee" containerName="config-reloader"
Sep 17 20:53:15.733209 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.733200    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="49b464f9-52b0-49b6-a3ff-a748723f65ee" containerName="prom-label-proxy"
Sep 17 20:53:15.738864 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.738832    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:15.741252 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.741233    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 17 20:53:15.741344 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.741325    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-d4z5z\""
Sep 17 20:53:15.741400 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.741365    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 17 20:53:15.741400 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.741378    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 17 20:53:15.741513 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.741464    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 17 20:53:15.741613 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.741592    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 17 20:53:15.741868 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.741685    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 17 20:53:15.741868 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.741737    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 17 20:53:15.742295 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.742279    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 17 20:53:15.746452 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.746434    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 17 20:53:15.753170 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.753146    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 20:53:15.885006 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.884978    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5ab4b357-f52a-4aa0-8cb2-27e6c5758243-web-config\") pod \"alertmanager-main-0\" (UID: \"5ab4b357-f52a-4aa0-8cb2-27e6c5758243\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:15.885006 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.885009    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ab4b357-f52a-4aa0-8cb2-27e6c5758243-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5ab4b357-f52a-4aa0-8cb2-27e6c5758243\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:15.885200 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.885034    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5ab4b357-f52a-4aa0-8cb2-27e6c5758243-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5ab4b357-f52a-4aa0-8cb2-27e6c5758243\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:15.885200 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.885077    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5ab4b357-f52a-4aa0-8cb2-27e6c5758243-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5ab4b357-f52a-4aa0-8cb2-27e6c5758243\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:15.885200 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.885120    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5ab4b357-f52a-4aa0-8cb2-27e6c5758243-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5ab4b357-f52a-4aa0-8cb2-27e6c5758243\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:15.885200 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.885161    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5ab4b357-f52a-4aa0-8cb2-27e6c5758243-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5ab4b357-f52a-4aa0-8cb2-27e6c5758243\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:15.885200 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.885179    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5ab4b357-f52a-4aa0-8cb2-27e6c5758243-config-out\") pod \"alertmanager-main-0\" (UID: \"5ab4b357-f52a-4aa0-8cb2-27e6c5758243\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:15.885348 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.885241    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ab4b357-f52a-4aa0-8cb2-27e6c5758243-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5ab4b357-f52a-4aa0-8cb2-27e6c5758243\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:15.885348 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.885261    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5ab4b357-f52a-4aa0-8cb2-27e6c5758243-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5ab4b357-f52a-4aa0-8cb2-27e6c5758243\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:15.885348 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.885281    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5ab4b357-f52a-4aa0-8cb2-27e6c5758243-config-volume\") pod \"alertmanager-main-0\" (UID: \"5ab4b357-f52a-4aa0-8cb2-27e6c5758243\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:15.885348 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.885301    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5ab4b357-f52a-4aa0-8cb2-27e6c5758243-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5ab4b357-f52a-4aa0-8cb2-27e6c5758243\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:15.885348 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.885322    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhwmr\" (UniqueName: \"kubernetes.io/projected/5ab4b357-f52a-4aa0-8cb2-27e6c5758243-kube-api-access-fhwmr\") pod \"alertmanager-main-0\" (UID: \"5ab4b357-f52a-4aa0-8cb2-27e6c5758243\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:15.885348 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.885344    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5ab4b357-f52a-4aa0-8cb2-27e6c5758243-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5ab4b357-f52a-4aa0-8cb2-27e6c5758243\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:15.985761 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.985713    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5ab4b357-f52a-4aa0-8cb2-27e6c5758243-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5ab4b357-f52a-4aa0-8cb2-27e6c5758243\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:15.985761 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.985759    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5ab4b357-f52a-4aa0-8cb2-27e6c5758243-config-out\") pod \"alertmanager-main-0\" (UID: \"5ab4b357-f52a-4aa0-8cb2-27e6c5758243\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:15.986058 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.985794    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ab4b357-f52a-4aa0-8cb2-27e6c5758243-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5ab4b357-f52a-4aa0-8cb2-27e6c5758243\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:15.986058 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.985812    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5ab4b357-f52a-4aa0-8cb2-27e6c5758243-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5ab4b357-f52a-4aa0-8cb2-27e6c5758243\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:15.986058 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.985948    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5ab4b357-f52a-4aa0-8cb2-27e6c5758243-config-volume\") pod \"alertmanager-main-0\" (UID: \"5ab4b357-f52a-4aa0-8cb2-27e6c5758243\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:15.986058 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.985991    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5ab4b357-f52a-4aa0-8cb2-27e6c5758243-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5ab4b357-f52a-4aa0-8cb2-27e6c5758243\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:15.986058 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.986038    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhwmr\" (UniqueName: \"kubernetes.io/projected/5ab4b357-f52a-4aa0-8cb2-27e6c5758243-kube-api-access-fhwmr\") pod \"alertmanager-main-0\" (UID: \"5ab4b357-f52a-4aa0-8cb2-27e6c5758243\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:15.986305 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.986080    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5ab4b357-f52a-4aa0-8cb2-27e6c5758243-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5ab4b357-f52a-4aa0-8cb2-27e6c5758243\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:15.986305 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.986133    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5ab4b357-f52a-4aa0-8cb2-27e6c5758243-web-config\") pod \"alertmanager-main-0\" (UID: \"5ab4b357-f52a-4aa0-8cb2-27e6c5758243\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:15.986305 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.986158    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ab4b357-f52a-4aa0-8cb2-27e6c5758243-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5ab4b357-f52a-4aa0-8cb2-27e6c5758243\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:15.986305 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.986190    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5ab4b357-f52a-4aa0-8cb2-27e6c5758243-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5ab4b357-f52a-4aa0-8cb2-27e6c5758243\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:15.986305 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.986224    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5ab4b357-f52a-4aa0-8cb2-27e6c5758243-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5ab4b357-f52a-4aa0-8cb2-27e6c5758243\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:15.987061 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.987035    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ab4b357-f52a-4aa0-8cb2-27e6c5758243-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5ab4b357-f52a-4aa0-8cb2-27e6c5758243\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:15.987255 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.987237    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5ab4b357-f52a-4aa0-8cb2-27e6c5758243-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5ab4b357-f52a-4aa0-8cb2-27e6c5758243\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:15.988067 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.987726    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ab4b357-f52a-4aa0-8cb2-27e6c5758243-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5ab4b357-f52a-4aa0-8cb2-27e6c5758243\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:15.989057 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.989030    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5ab4b357-f52a-4aa0-8cb2-27e6c5758243-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5ab4b357-f52a-4aa0-8cb2-27e6c5758243\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:15.989173 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.989063    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5ab4b357-f52a-4aa0-8cb2-27e6c5758243-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5ab4b357-f52a-4aa0-8cb2-27e6c5758243\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:15.989173 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.989115    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5ab4b357-f52a-4aa0-8cb2-27e6c5758243-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5ab4b357-f52a-4aa0-8cb2-27e6c5758243\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:15.989173 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.989139    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5ab4b357-f52a-4aa0-8cb2-27e6c5758243-config-volume\") pod \"alertmanager-main-0\" (UID: \"5ab4b357-f52a-4aa0-8cb2-27e6c5758243\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:15.989173 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.989154    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5ab4b357-f52a-4aa0-8cb2-27e6c5758243-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5ab4b357-f52a-4aa0-8cb2-27e6c5758243\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:15.989772 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.989742    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5ab4b357-f52a-4aa0-8cb2-27e6c5758243-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5ab4b357-f52a-4aa0-8cb2-27e6c5758243\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:15.989876 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.989785    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5ab4b357-f52a-4aa0-8cb2-27e6c5758243-config-out\") pod \"alertmanager-main-0\" (UID: \"5ab4b357-f52a-4aa0-8cb2-27e6c5758243\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:15.990015 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.989994    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5ab4b357-f52a-4aa0-8cb2-27e6c5758243-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5ab4b357-f52a-4aa0-8cb2-27e6c5758243\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:15.990256 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.990239    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5ab4b357-f52a-4aa0-8cb2-27e6c5758243-web-config\") pod \"alertmanager-main-0\" (UID: \"5ab4b357-f52a-4aa0-8cb2-27e6c5758243\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:15.991186 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.991168    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5ab4b357-f52a-4aa0-8cb2-27e6c5758243-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5ab4b357-f52a-4aa0-8cb2-27e6c5758243\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:15.998013 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:15.997987    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhwmr\" (UniqueName: \"kubernetes.io/projected/5ab4b357-f52a-4aa0-8cb2-27e6c5758243-kube-api-access-fhwmr\") pod \"alertmanager-main-0\" (UID: \"5ab4b357-f52a-4aa0-8cb2-27e6c5758243\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:16.050016 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:16.049983    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:53:16.330019 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:16.329981    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 20:53:16.331323 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:53:16.331300    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ab4b357_f52a_4aa0_8cb2_27e6c5758243.slice/crio-5423dbafb0664d5b3bfa8e29d54fe6f4723c722fa74f17ec7d283ab6c0115fc6 WatchSource:0}: Error finding container 5423dbafb0664d5b3bfa8e29d54fe6f4723c722fa74f17ec7d283ab6c0115fc6: Status 404 returned error can't find the container with id 5423dbafb0664d5b3bfa8e29d54fe6f4723c722fa74f17ec7d283ab6c0115fc6
Sep 17 20:53:16.380363 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:16.380333    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5ab4b357-f52a-4aa0-8cb2-27e6c5758243","Type":"ContainerStarted","Data":"5423dbafb0664d5b3bfa8e29d54fe6f4723c722fa74f17ec7d283ab6c0115fc6"}
Sep 17 20:53:17.355691 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:17.355651    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49b464f9-52b0-49b6-a3ff-a748723f65ee" path="/var/lib/kubelet/pods/49b464f9-52b0-49b6-a3ff-a748723f65ee/volumes"
Sep 17 20:53:17.385663 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:17.385626    2575 generic.go:356] "Generic (PLEG): container finished" podID="5ab4b357-f52a-4aa0-8cb2-27e6c5758243" containerID="4a8654f64e325426b17dc84bce4ce35aaba84f3d30c05b9ac1cc8ccdae4db26b" exitCode=0
Sep 17 20:53:17.385825 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:17.385710    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5ab4b357-f52a-4aa0-8cb2-27e6c5758243","Type":"ContainerDied","Data":"4a8654f64e325426b17dc84bce4ce35aaba84f3d30c05b9ac1cc8ccdae4db26b"}
Sep 17 20:53:18.097185 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:18.097144    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-685dfdf6c8-5n6gw"]
Sep 17 20:53:18.100863 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:18.100823    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-685dfdf6c8-5n6gw"
Sep 17 20:53:18.103029 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:18.102995    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-lv8hz\""
Sep 17 20:53:18.103158 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:18.103132    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 17 20:53:18.103221 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:18.103213    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 17 20:53:18.103297 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:18.103266    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 17 20:53:18.103354 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:18.103273    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 17 20:53:18.103807 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:18.103750    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 17 20:53:18.109400 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:18.109371    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 17 20:53:18.116799 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:18.116766    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-685dfdf6c8-5n6gw"]
Sep 17 20:53:18.206383 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:18.206347    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4685e11-32bc-4200-8737-4dfecd8f7698-telemeter-trusted-ca-bundle\") pod \"telemeter-client-685dfdf6c8-5n6gw\" (UID: \"b4685e11-32bc-4200-8737-4dfecd8f7698\") " pod="openshift-monitoring/telemeter-client-685dfdf6c8-5n6gw"
Sep 17 20:53:18.206383 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:18.206387    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b4685e11-32bc-4200-8737-4dfecd8f7698-secret-telemeter-client\") pod \"telemeter-client-685dfdf6c8-5n6gw\" (UID: \"b4685e11-32bc-4200-8737-4dfecd8f7698\") " pod="openshift-monitoring/telemeter-client-685dfdf6c8-5n6gw"
Sep 17 20:53:18.206634 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:18.206463    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b4685e11-32bc-4200-8737-4dfecd8f7698-metrics-client-ca\") pod \"telemeter-client-685dfdf6c8-5n6gw\" (UID: \"b4685e11-32bc-4200-8737-4dfecd8f7698\") " pod="openshift-monitoring/telemeter-client-685dfdf6c8-5n6gw"
Sep 17 20:53:18.206634 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:18.206496    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b4685e11-32bc-4200-8737-4dfecd8f7698-federate-client-tls\") pod \"telemeter-client-685dfdf6c8-5n6gw\" (UID: \"b4685e11-32bc-4200-8737-4dfecd8f7698\") " pod="openshift-monitoring/telemeter-client-685dfdf6c8-5n6gw"
Sep 17 20:53:18.206634 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:18.206514    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b4685e11-32bc-4200-8737-4dfecd8f7698-telemeter-client-tls\") pod \"telemeter-client-685dfdf6c8-5n6gw\" (UID: \"b4685e11-32bc-4200-8737-4dfecd8f7698\") " pod="openshift-monitoring/telemeter-client-685dfdf6c8-5n6gw"
Sep 17 20:53:18.206634 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:18.206530    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b4685e11-32bc-4200-8737-4dfecd8f7698-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-685dfdf6c8-5n6gw\" (UID: \"b4685e11-32bc-4200-8737-4dfecd8f7698\") " pod="openshift-monitoring/telemeter-client-685dfdf6c8-5n6gw"
Sep 17 20:53:18.206634 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:18.206559    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4685e11-32bc-4200-8737-4dfecd8f7698-serving-certs-ca-bundle\") pod \"telemeter-client-685dfdf6c8-5n6gw\" (UID: \"b4685e11-32bc-4200-8737-4dfecd8f7698\") " pod="openshift-monitoring/telemeter-client-685dfdf6c8-5n6gw"
Sep 17 20:53:18.206634 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:18.206601    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrzgn\" (UniqueName: \"kubernetes.io/projected/b4685e11-32bc-4200-8737-4dfecd8f7698-kube-api-access-hrzgn\") pod \"telemeter-client-685dfdf6c8-5n6gw\" (UID: \"b4685e11-32bc-4200-8737-4dfecd8f7698\") " pod="openshift-monitoring/telemeter-client-685dfdf6c8-5n6gw"
Sep 17 20:53:18.307183 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:18.307138    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4685e11-32bc-4200-8737-4dfecd8f7698-telemeter-trusted-ca-bundle\") pod \"telemeter-client-685dfdf6c8-5n6gw\" (UID: \"b4685e11-32bc-4200-8737-4dfecd8f7698\") " pod="openshift-monitoring/telemeter-client-685dfdf6c8-5n6gw"
Sep 17 20:53:18.308424 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:18.308393    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b4685e11-32bc-4200-8737-4dfecd8f7698-secret-telemeter-client\") pod \"telemeter-client-685dfdf6c8-5n6gw\" (UID: \"b4685e11-32bc-4200-8737-4dfecd8f7698\") " pod="openshift-monitoring/telemeter-client-685dfdf6c8-5n6gw"
Sep 17 20:53:18.308570 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:18.308493    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b4685e11-32bc-4200-8737-4dfecd8f7698-metrics-client-ca\") pod \"telemeter-client-685dfdf6c8-5n6gw\" (UID: \"b4685e11-32bc-4200-8737-4dfecd8f7698\") " pod="openshift-monitoring/telemeter-client-685dfdf6c8-5n6gw"
Sep 17 20:53:18.308570 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:18.308539    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b4685e11-32bc-4200-8737-4dfecd8f7698-federate-client-tls\") pod \"telemeter-client-685dfdf6c8-5n6gw\" (UID: \"b4685e11-32bc-4200-8737-4dfecd8f7698\") " pod="openshift-monitoring/telemeter-client-685dfdf6c8-5n6gw"
Sep 17 20:53:18.308570 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:18.308563    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b4685e11-32bc-4200-8737-4dfecd8f7698-telemeter-client-tls\") pod \"telemeter-client-685dfdf6c8-5n6gw\" (UID: \"b4685e11-32bc-4200-8737-4dfecd8f7698\") " pod="openshift-monitoring/telemeter-client-685dfdf6c8-5n6gw"
Sep 17 20:53:18.308729 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:18.308589    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b4685e11-32bc-4200-8737-4dfecd8f7698-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-685dfdf6c8-5n6gw\" (UID: \"b4685e11-32bc-4200-8737-4dfecd8f7698\") " pod="openshift-monitoring/telemeter-client-685dfdf6c8-5n6gw"
Sep 17 20:53:18.308729 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:18.308334    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4685e11-32bc-4200-8737-4dfecd8f7698-telemeter-trusted-ca-bundle\") pod \"telemeter-client-685dfdf6c8-5n6gw\" (UID: \"b4685e11-32bc-4200-8737-4dfecd8f7698\") " pod="openshift-monitoring/telemeter-client-685dfdf6c8-5n6gw"
Sep 17 20:53:18.308729 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:18.308627    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4685e11-32bc-4200-8737-4dfecd8f7698-serving-certs-ca-bundle\") pod \"telemeter-client-685dfdf6c8-5n6gw\" (UID: \"b4685e11-32bc-4200-8737-4dfecd8f7698\") " pod="openshift-monitoring/telemeter-client-685dfdf6c8-5n6gw"
Sep 17 20:53:18.308729 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:18.308701    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrzgn\" (UniqueName: \"kubernetes.io/projected/b4685e11-32bc-4200-8737-4dfecd8f7698-kube-api-access-hrzgn\") pod \"telemeter-client-685dfdf6c8-5n6gw\" (UID: \"b4685e11-32bc-4200-8737-4dfecd8f7698\") " pod="openshift-monitoring/telemeter-client-685dfdf6c8-5n6gw"
Sep 17 20:53:18.309319 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:18.309270    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b4685e11-32bc-4200-8737-4dfecd8f7698-metrics-client-ca\") pod \"telemeter-client-685dfdf6c8-5n6gw\" (UID: \"b4685e11-32bc-4200-8737-4dfecd8f7698\") " pod="openshift-monitoring/telemeter-client-685dfdf6c8-5n6gw"
Sep 17 20:53:18.309503 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:18.309481    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4685e11-32bc-4200-8737-4dfecd8f7698-serving-certs-ca-bundle\") pod \"telemeter-client-685dfdf6c8-5n6gw\" (UID: \"b4685e11-32bc-4200-8737-4dfecd8f7698\") " pod="openshift-monitoring/telemeter-client-685dfdf6c8-5n6gw"
Sep 17 20:53:18.311557 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:18.311530    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b4685e11-32bc-4200-8737-4dfecd8f7698-federate-client-tls\") pod \"telemeter-client-685dfdf6c8-5n6gw\" (UID: \"b4685e11-32bc-4200-8737-4dfecd8f7698\") " pod="openshift-monitoring/telemeter-client-685dfdf6c8-5n6gw"
Sep 17 20:53:18.311663 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:18.311600    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b4685e11-32bc-4200-8737-4dfecd8f7698-telemeter-client-tls\") pod \"telemeter-client-685dfdf6c8-5n6gw\" (UID: \"b4685e11-32bc-4200-8737-4dfecd8f7698\") " pod="openshift-monitoring/telemeter-client-685dfdf6c8-5n6gw"
Sep 17 20:53:18.311733 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:18.311655    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b4685e11-32bc-4200-8737-4dfecd8f7698-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-685dfdf6c8-5n6gw\" (UID: \"b4685e11-32bc-4200-8737-4dfecd8f7698\") " pod="openshift-monitoring/telemeter-client-685dfdf6c8-5n6gw"
Sep 17 20:53:18.312129 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:18.312110    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b4685e11-32bc-4200-8737-4dfecd8f7698-secret-telemeter-client\") pod \"telemeter-client-685dfdf6c8-5n6gw\" (UID: \"b4685e11-32bc-4200-8737-4dfecd8f7698\") " pod="openshift-monitoring/telemeter-client-685dfdf6c8-5n6gw"
Sep 17 20:53:18.317968 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:18.317946    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrzgn\" (UniqueName: \"kubernetes.io/projected/b4685e11-32bc-4200-8737-4dfecd8f7698-kube-api-access-hrzgn\") pod \"telemeter-client-685dfdf6c8-5n6gw\" (UID: \"b4685e11-32bc-4200-8737-4dfecd8f7698\") " pod="openshift-monitoring/telemeter-client-685dfdf6c8-5n6gw"
Sep 17 20:53:18.394710 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:18.394626    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5ab4b357-f52a-4aa0-8cb2-27e6c5758243","Type":"ContainerStarted","Data":"fc6b9d451ba54798afba6dfadb5584e8b12361eae543b0c96afecc438cb86763"}
Sep 17 20:53:18.394710 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:18.394695    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5ab4b357-f52a-4aa0-8cb2-27e6c5758243","Type":"ContainerStarted","Data":"5d8fdc57bc60009f4d8256c4457123827ee40cbb131a7d056ff2a0736cfec97d"}
Sep 17 20:53:18.395210 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:18.394723    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5ab4b357-f52a-4aa0-8cb2-27e6c5758243","Type":"ContainerStarted","Data":"f5dd973c680f8c6991d4101d7e3155ee55c542033108b949b2d022cde64f1b49"}
Sep 17 20:53:18.395210 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:18.394761    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5ab4b357-f52a-4aa0-8cb2-27e6c5758243","Type":"ContainerStarted","Data":"59957ba29ba4d15e83c6ecba3a70367e7e396b8be165a3de902a659adbc73e2f"}
Sep 17 20:53:18.395210 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:18.394803    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5ab4b357-f52a-4aa0-8cb2-27e6c5758243","Type":"ContainerStarted","Data":"4e7c2077f2b1fa227b88db5d09b6b1ba5ba2c8fc83b0d5ba8c88508ad6dc8568"}
Sep 17 20:53:18.395210 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:18.394859    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5ab4b357-f52a-4aa0-8cb2-27e6c5758243","Type":"ContainerStarted","Data":"bac0ec84b63830cc359a7acafd1c48f3f40059652c932a8ebc72228a4faf9b07"}
Sep 17 20:53:18.412338 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:18.412308    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-685dfdf6c8-5n6gw"
Sep 17 20:53:18.424386 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:18.424315    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.408496181 podStartE2EDuration="3.424291314s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.795133ms" isStatefulPod=false podCreationTimestamp="2026-09-17 20:53:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:53:18.42221598 +0000 UTC m=+263.574660377" watchObservedRunningTime="2026-09-17 20:53:18.424291314 +0000 UTC m=+263.576735693"
Sep 17 20:53:18.706426 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:18.706061    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-685dfdf6c8-5n6gw"]
Sep 17 20:53:18.708730 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:53:18.708697    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4685e11_32bc_4200_8737_4dfecd8f7698.slice/crio-013e0249e00190dec600d8f4f6033ce80020353d4f192151aa14fbbca4a13482 WatchSource:0}: Error finding container 013e0249e00190dec600d8f4f6033ce80020353d4f192151aa14fbbca4a13482: Status 404 returned error can't find the container with id 013e0249e00190dec600d8f4f6033ce80020353d4f192151aa14fbbca4a13482
Sep 17 20:53:19.400236 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:19.400201    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-685dfdf6c8-5n6gw" event={"ID":"b4685e11-32bc-4200-8737-4dfecd8f7698","Type":"ContainerStarted","Data":"013e0249e00190dec600d8f4f6033ce80020353d4f192151aa14fbbca4a13482"}
Sep 17 20:53:21.411306 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:21.411270    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-685dfdf6c8-5n6gw" event={"ID":"b4685e11-32bc-4200-8737-4dfecd8f7698","Type":"ContainerStarted","Data":"ccc78f00d22b626ae37d2a9786c4618dbad166b241e571d743a6a5152ed48513"}
Sep 17 20:53:21.411680 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:21.411330    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-685dfdf6c8-5n6gw" event={"ID":"b4685e11-32bc-4200-8737-4dfecd8f7698","Type":"ContainerStarted","Data":"57ee49406dd7755137581bbf4394075445622bad56ab6abbbdd3866968c1ae5b"}
Sep 17 20:53:21.411680 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:21.411350    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-685dfdf6c8-5n6gw" event={"ID":"b4685e11-32bc-4200-8737-4dfecd8f7698","Type":"ContainerStarted","Data":"07645345118ca3d2a8fc1bd3bb7b597d858081d533f2bd86c9ab1dd792f476e6"}
Sep 17 20:53:21.436056 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:21.436005    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-685dfdf6c8-5n6gw" podStartSLOduration=1.61384858 podStartE2EDuration="3.435988137s" totalImagesPullingTime="1.822139557s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:53:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:53:21.434522189 +0000 UTC m=+266.586966567" watchObservedRunningTime="2026-09-17 20:53:21.435988137 +0000 UTC m=+266.588432512"
Sep 17 20:53:21.588568 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:21.588530    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:53:21.644265 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:21.644232    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:53:25.782674 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:25.782640    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:53:25.883769 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:25.883737    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:53:32.860572 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:32.860529    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:53:32.961493 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:32.961443    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:53:33.022235 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:33.022207    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:53:34.839810 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:34.839775    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:53:34.936349 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:34.936319    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:53:45.336669 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:45.336637    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:53:45.403493 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:45.403457    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:53:55.276307 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:53:55.276283    2575 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 20:54:28.275564 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:54:28.275526    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:54:28.336304 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:54:28.336261    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:54:28.571276 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:54:28.571183    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:54:28.647521 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:54:28.647483    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:54:28.705627 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:54:28.705595    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:54:29.286126 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:54:29.286089    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:54:29.348678 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:54:29.348650    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:54:39.082504 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:54:39.082473    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:54:39.180395 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:54:39.180367    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:54:39.240997 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:54:39.240961    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:54:40.317681 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:54:40.317643    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:54:40.378108 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:54:40.378079    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:54:55.551370 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:54:55.551339    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:54:55.656352 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:54:55.656315    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:55:32.650284 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:55:32.650250    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:55:32.737736 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:55:32.737703    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:55:45.068624 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:55:45.068593    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:55:45.168375 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:55:45.168337    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:55:45.267439 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:55:45.267404    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:55:59.574469 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:55:59.574405    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:55:59.631246 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:55:59.631203    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:56:07.528775 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:07.528724    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:56:08.784643 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:08.784602    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:56:08.840586 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:08.840547    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-rt7st"]
Sep 17 20:56:08.843800 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:08.843779    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-rt7st"
Sep 17 20:56:08.846454 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:08.846431    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 17 20:56:08.846572 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:08.846517    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 17 20:56:08.846623 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:08.846609    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-w8hxt\""
Sep 17 20:56:08.846674 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:08.846665    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 17 20:56:08.856898 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:08.856868    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-rt7st"]
Sep 17 20:56:08.981082 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:08.981049    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/011656d3-2e8e-45ba-9a5c-d365cc6128ed-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-rt7st\" (UID: \"011656d3-2e8e-45ba-9a5c-d365cc6128ed\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-rt7st"
Sep 17 20:56:08.981226 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:08.981102    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6qqc\" (UniqueName: \"kubernetes.io/projected/011656d3-2e8e-45ba-9a5c-d365cc6128ed-kube-api-access-s6qqc\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-rt7st\" (UID: \"011656d3-2e8e-45ba-9a5c-d365cc6128ed\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-rt7st"
Sep 17 20:56:09.081948 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:09.081866    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/011656d3-2e8e-45ba-9a5c-d365cc6128ed-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-rt7st\" (UID: \"011656d3-2e8e-45ba-9a5c-d365cc6128ed\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-rt7st"
Sep 17 20:56:09.081948 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:09.081929    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6qqc\" (UniqueName: \"kubernetes.io/projected/011656d3-2e8e-45ba-9a5c-d365cc6128ed-kube-api-access-s6qqc\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-rt7st\" (UID: \"011656d3-2e8e-45ba-9a5c-d365cc6128ed\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-rt7st"
Sep 17 20:56:09.084228 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:09.084204    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/011656d3-2e8e-45ba-9a5c-d365cc6128ed-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-rt7st\" (UID: \"011656d3-2e8e-45ba-9a5c-d365cc6128ed\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-rt7st"
Sep 17 20:56:09.090678 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:09.090658    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6qqc\" (UniqueName: \"kubernetes.io/projected/011656d3-2e8e-45ba-9a5c-d365cc6128ed-kube-api-access-s6qqc\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-rt7st\" (UID: \"011656d3-2e8e-45ba-9a5c-d365cc6128ed\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-rt7st"
Sep 17 20:56:09.154385 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:09.154351    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-rt7st"
Sep 17 20:56:09.435371 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:09.435287    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-rt7st"]
Sep 17 20:56:09.439693 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:56:09.439659    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod011656d3_2e8e_45ba_9a5c_d365cc6128ed.slice/crio-8223e1b72a1038d0d470fbf611f05320a573a5ef19f91f0aef0d54c639fe3905 WatchSource:0}: Error finding container 8223e1b72a1038d0d470fbf611f05320a573a5ef19f91f0aef0d54c639fe3905: Status 404 returned error can't find the container with id 8223e1b72a1038d0d470fbf611f05320a573a5ef19f91f0aef0d54c639fe3905
Sep 17 20:56:09.441572 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:09.441549    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 20:56:10.074430 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:10.074391    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-rt7st" event={"ID":"011656d3-2e8e-45ba-9a5c-d365cc6128ed","Type":"ContainerStarted","Data":"8223e1b72a1038d0d470fbf611f05320a573a5ef19f91f0aef0d54c639fe3905"}
Sep 17 20:56:12.990327 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:12.990290    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:56:13.090677 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:13.090644    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-rt7st" event={"ID":"011656d3-2e8e-45ba-9a5c-d365cc6128ed","Type":"ContainerStarted","Data":"668d3cd046bb1b8561f49faff9a741b7d9689db3b0884c2c864579c8bdca710a"}
Sep 17 20:56:13.090864 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:13.090820    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-rt7st"
Sep 17 20:56:13.112614 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:13.112565    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-rt7st" podStartSLOduration=2.117007538 podStartE2EDuration="5.112549511s" totalImagesPullingTime="2.995541973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:56:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:56:13.10992983 +0000 UTC m=+438.262374203" watchObservedRunningTime="2026-09-17 20:56:13.112549511 +0000 UTC m=+438.264993887"
Sep 17 20:56:13.271332 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:13.271259    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-52h5t"]
Sep 17 20:56:13.273864 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:13.273823    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:56:13.274697 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:13.274682    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-52h5t"
Sep 17 20:56:13.277004 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:13.276973    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-ld964\""
Sep 17 20:56:13.277107 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:13.277068    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 17 20:56:13.277225 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:13.277210    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 17 20:56:13.283958 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:13.283935    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-52h5t"]
Sep 17 20:56:13.401302 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:13.401262    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:56:13.419698 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:13.419665    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/6cea2f99-3bea-4cdf-a9d9-4596ba4546e5-cabundle0\") pod \"keda-operator-579c8958f7-52h5t\" (UID: \"6cea2f99-3bea-4cdf-a9d9-4596ba4546e5\") " pod="openshift-keda/keda-operator-579c8958f7-52h5t"
Sep 17 20:56:13.419891 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:13.419725    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg9qr\" (UniqueName: \"kubernetes.io/projected/6cea2f99-3bea-4cdf-a9d9-4596ba4546e5-kube-api-access-fg9qr\") pod \"keda-operator-579c8958f7-52h5t\" (UID: \"6cea2f99-3bea-4cdf-a9d9-4596ba4546e5\") " pod="openshift-keda/keda-operator-579c8958f7-52h5t"
Sep 17 20:56:13.419891 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:13.419806    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/6cea2f99-3bea-4cdf-a9d9-4596ba4546e5-certificates\") pod \"keda-operator-579c8958f7-52h5t\" (UID: \"6cea2f99-3bea-4cdf-a9d9-4596ba4546e5\") " pod="openshift-keda/keda-operator-579c8958f7-52h5t"
Sep 17 20:56:13.478609 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:13.478577    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-lcjld"]
Sep 17 20:56:13.479437 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:13.479405    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:56:13.482181 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:13.482165    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-lcjld"
Sep 17 20:56:13.484701 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:13.484682    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Sep 17 20:56:13.491305 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:13.491278    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-lcjld"]
Sep 17 20:56:13.520965 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:13.520934    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg9qr\" (UniqueName: \"kubernetes.io/projected/6cea2f99-3bea-4cdf-a9d9-4596ba4546e5-kube-api-access-fg9qr\") pod \"keda-operator-579c8958f7-52h5t\" (UID: \"6cea2f99-3bea-4cdf-a9d9-4596ba4546e5\") " pod="openshift-keda/keda-operator-579c8958f7-52h5t"
Sep 17 20:56:13.521128 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:13.520995    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/6cea2f99-3bea-4cdf-a9d9-4596ba4546e5-certificates\") pod \"keda-operator-579c8958f7-52h5t\" (UID: \"6cea2f99-3bea-4cdf-a9d9-4596ba4546e5\") " pod="openshift-keda/keda-operator-579c8958f7-52h5t"
Sep 17 20:56:13.521128 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:13.521064    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/6cea2f99-3bea-4cdf-a9d9-4596ba4546e5-cabundle0\") pod \"keda-operator-579c8958f7-52h5t\" (UID: \"6cea2f99-3bea-4cdf-a9d9-4596ba4546e5\") " pod="openshift-keda/keda-operator-579c8958f7-52h5t"
Sep 17 20:56:13.521250 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:56:13.521133    2575 secret.go:281] references non-existent secret key: ca.crt
Sep 17 20:56:13.521250 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:56:13.521156    2575 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 17 20:56:13.521250 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:56:13.521167    2575 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-52h5t: references non-existent secret key: ca.crt
Sep 17 20:56:13.521250 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:56:13.521244    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6cea2f99-3bea-4cdf-a9d9-4596ba4546e5-certificates podName:6cea2f99-3bea-4cdf-a9d9-4596ba4546e5 nodeName:}" failed. No retries permitted until 2026-09-17 20:56:14.021222675 +0000 UTC m=+439.173667042 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/6cea2f99-3bea-4cdf-a9d9-4596ba4546e5-certificates") pod "keda-operator-579c8958f7-52h5t" (UID: "6cea2f99-3bea-4cdf-a9d9-4596ba4546e5") : references non-existent secret key: ca.crt
Sep 17 20:56:13.521787 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:13.521732    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/6cea2f99-3bea-4cdf-a9d9-4596ba4546e5-cabundle0\") pod \"keda-operator-579c8958f7-52h5t\" (UID: \"6cea2f99-3bea-4cdf-a9d9-4596ba4546e5\") " pod="openshift-keda/keda-operator-579c8958f7-52h5t"
Sep 17 20:56:13.537390 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:13.531712    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg9qr\" (UniqueName: \"kubernetes.io/projected/6cea2f99-3bea-4cdf-a9d9-4596ba4546e5-kube-api-access-fg9qr\") pod \"keda-operator-579c8958f7-52h5t\" (UID: \"6cea2f99-3bea-4cdf-a9d9-4596ba4546e5\") " pod="openshift-keda/keda-operator-579c8958f7-52h5t"
Sep 17 20:56:13.621632 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:13.621587    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/33b379dc-8ca9-4de9-a7f6-82fe14cb2472-certificates\") pod \"keda-metrics-apiserver-77857b7b98-lcjld\" (UID: \"33b379dc-8ca9-4de9-a7f6-82fe14cb2472\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-lcjld"
Sep 17 20:56:13.621811 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:13.621659    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/33b379dc-8ca9-4de9-a7f6-82fe14cb2472-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-lcjld\" (UID: \"33b379dc-8ca9-4de9-a7f6-82fe14cb2472\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-lcjld"
Sep 17 20:56:13.621811 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:13.621784    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk2nt\" (UniqueName: \"kubernetes.io/projected/33b379dc-8ca9-4de9-a7f6-82fe14cb2472-kube-api-access-nk2nt\") pod \"keda-metrics-apiserver-77857b7b98-lcjld\" (UID: \"33b379dc-8ca9-4de9-a7f6-82fe14cb2472\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-lcjld"
Sep 17 20:56:13.722916 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:13.722886    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/33b379dc-8ca9-4de9-a7f6-82fe14cb2472-certificates\") pod \"keda-metrics-apiserver-77857b7b98-lcjld\" (UID: \"33b379dc-8ca9-4de9-a7f6-82fe14cb2472\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-lcjld"
Sep 17 20:56:13.723097 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:13.722927    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/33b379dc-8ca9-4de9-a7f6-82fe14cb2472-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-lcjld\" (UID: \"33b379dc-8ca9-4de9-a7f6-82fe14cb2472\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-lcjld"
Sep 17 20:56:13.723097 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:13.722984    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk2nt\" (UniqueName: \"kubernetes.io/projected/33b379dc-8ca9-4de9-a7f6-82fe14cb2472-kube-api-access-nk2nt\") pod \"keda-metrics-apiserver-77857b7b98-lcjld\" (UID: \"33b379dc-8ca9-4de9-a7f6-82fe14cb2472\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-lcjld"
Sep 17 20:56:13.723097 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:56:13.723057    2575 projected.go:266] Couldn't get secret openshift-keda/keda-metrics-apiserver-certs: secret "keda-metrics-apiserver-certs" not found
Sep 17 20:56:13.723097 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:56:13.723082    2575 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-77857b7b98-lcjld: secret "keda-metrics-apiserver-certs" not found
Sep 17 20:56:13.723239 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:56:13.723143    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33b379dc-8ca9-4de9-a7f6-82fe14cb2472-certificates podName:33b379dc-8ca9-4de9-a7f6-82fe14cb2472 nodeName:}" failed. No retries permitted until 2026-09-17 20:56:14.223127032 +0000 UTC m=+439.375571385 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/33b379dc-8ca9-4de9-a7f6-82fe14cb2472-certificates") pod "keda-metrics-apiserver-77857b7b98-lcjld" (UID: "33b379dc-8ca9-4de9-a7f6-82fe14cb2472") : secret "keda-metrics-apiserver-certs" not found
Sep 17 20:56:13.723462 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:13.723441    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/33b379dc-8ca9-4de9-a7f6-82fe14cb2472-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-lcjld\" (UID: \"33b379dc-8ca9-4de9-a7f6-82fe14cb2472\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-lcjld"
Sep 17 20:56:13.733518 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:13.733468    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk2nt\" (UniqueName: \"kubernetes.io/projected/33b379dc-8ca9-4de9-a7f6-82fe14cb2472-kube-api-access-nk2nt\") pod \"keda-metrics-apiserver-77857b7b98-lcjld\" (UID: \"33b379dc-8ca9-4de9-a7f6-82fe14cb2472\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-lcjld"
Sep 17 20:56:13.781480 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:13.781392    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:56:13.878418 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:13.878377    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:56:13.989730 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:13.989687    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:56:14.025306 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:14.025271    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/6cea2f99-3bea-4cdf-a9d9-4596ba4546e5-certificates\") pod \"keda-operator-579c8958f7-52h5t\" (UID: \"6cea2f99-3bea-4cdf-a9d9-4596ba4546e5\") " pod="openshift-keda/keda-operator-579c8958f7-52h5t"
Sep 17 20:56:14.027672 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:14.027649    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/6cea2f99-3bea-4cdf-a9d9-4596ba4546e5-certificates\") pod \"keda-operator-579c8958f7-52h5t\" (UID: \"6cea2f99-3bea-4cdf-a9d9-4596ba4546e5\") " pod="openshift-keda/keda-operator-579c8958f7-52h5t"
Sep 17 20:56:14.179129 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:14.179035    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:56:14.186003 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:14.185972    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-52h5t"
Sep 17 20:56:14.227471 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:14.227380    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/33b379dc-8ca9-4de9-a7f6-82fe14cb2472-certificates\") pod \"keda-metrics-apiserver-77857b7b98-lcjld\" (UID: \"33b379dc-8ca9-4de9-a7f6-82fe14cb2472\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-lcjld"
Sep 17 20:56:14.230152 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:14.230122    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/33b379dc-8ca9-4de9-a7f6-82fe14cb2472-certificates\") pod \"keda-metrics-apiserver-77857b7b98-lcjld\" (UID: \"33b379dc-8ca9-4de9-a7f6-82fe14cb2472\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-lcjld"
Sep 17 20:56:14.392931 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:14.392896    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-lcjld"
Sep 17 20:56:14.396527 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:14.396497    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:56:14.462618 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:14.462566    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-52h5t"]
Sep 17 20:56:14.466624 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:56:14.466594    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cea2f99_3bea_4cdf_a9d9_4596ba4546e5.slice/crio-5eba849daef567edfbfecbcaa8da91fd7f56a3eca0a72ba38596e44acbd35b40 WatchSource:0}: Error finding container 5eba849daef567edfbfecbcaa8da91fd7f56a3eca0a72ba38596e44acbd35b40: Status 404 returned error can't find the container with id 5eba849daef567edfbfecbcaa8da91fd7f56a3eca0a72ba38596e44acbd35b40
Sep 17 20:56:14.551455 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:14.551416    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:56:14.649902 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:14.649866    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:56:14.670456 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:14.670417    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-lcjld"]
Sep 17 20:56:14.672981 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:56:14.672954    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33b379dc_8ca9_4de9_a7f6_82fe14cb2472.slice/crio-cd268562b50ba25b1a9b9948280da7ea9c0c7e182fb61338381adcc6f20c3924 WatchSource:0}: Error finding container cd268562b50ba25b1a9b9948280da7ea9c0c7e182fb61338381adcc6f20c3924: Status 404 returned error can't find the container with id cd268562b50ba25b1a9b9948280da7ea9c0c7e182fb61338381adcc6f20c3924
Sep 17 20:56:15.102882 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:15.102821    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-lcjld" event={"ID":"33b379dc-8ca9-4de9-a7f6-82fe14cb2472","Type":"ContainerStarted","Data":"cd268562b50ba25b1a9b9948280da7ea9c0c7e182fb61338381adcc6f20c3924"}
Sep 17 20:56:15.104558 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:15.104526    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-52h5t" event={"ID":"6cea2f99-3bea-4cdf-a9d9-4596ba4546e5","Type":"ContainerStarted","Data":"5eba849daef567edfbfecbcaa8da91fd7f56a3eca0a72ba38596e44acbd35b40"}
Sep 17 20:56:19.131193 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:19.131143    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-lcjld" event={"ID":"33b379dc-8ca9-4de9-a7f6-82fe14cb2472","Type":"ContainerStarted","Data":"5eac9ffaec1aecf7135489ebd94978c4a902dddf2aa3861724e869a359e5a1d5"}
Sep 17 20:56:19.131660 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:19.131347    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-lcjld"
Sep 17 20:56:19.132818 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:19.132792    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-52h5t" event={"ID":"6cea2f99-3bea-4cdf-a9d9-4596ba4546e5","Type":"ContainerStarted","Data":"cdf13df385b68e1105386949d1ee58899e13847f7282c7ddcbbcfc2d0f91dd97"}
Sep 17 20:56:19.132942 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:19.132832    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-52h5t"
Sep 17 20:56:19.150003 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:19.149958    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-lcjld" podStartSLOduration=2.747368138 podStartE2EDuration="6.149944324s" totalImagesPullingTime="3.402576186s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:56:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:56:19.149095119 +0000 UTC m=+444.301539495" watchObservedRunningTime="2026-09-17 20:56:19.149944324 +0000 UTC m=+444.302388693"
Sep 17 20:56:19.170329 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:19.170279    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-52h5t" podStartSLOduration=2.556910562 podStartE2EDuration="6.170259632s" totalImagesPullingTime="3.61334907s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:56:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:56:19.169488611 +0000 UTC m=+444.321932988" watchObservedRunningTime="2026-09-17 20:56:19.170259632 +0000 UTC m=+444.322704009"
Sep 17 20:56:24.553461 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:24.553429    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:56:24.633661 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:24.633626    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:56:30.140930 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:30.140900    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-lcjld"
Sep 17 20:56:34.098968 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:34.098934    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-rt7st"
Sep 17 20:56:40.138397 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:56:40.138368    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-52h5t"
Sep 17 20:57:07.146894 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:07.146836    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:57:07.247655 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:07.247614    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:57:11.776290 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:11.776247    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:57:11.835630 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:11.835598    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:57:18.324057 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:18.324011    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:57:19.472340 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:19.472300    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:57:21.143581 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:21.143547    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-pmm4f"]
Sep 17 20:57:21.146990 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:21.146972    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-pmm4f"
Sep 17 20:57:21.150947 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:21.150927    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 17 20:57:21.151196 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:21.151173    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 17 20:57:21.151249 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:21.151209    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-fspg6\""
Sep 17 20:57:21.168970 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:21.168939    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-pmm4f"]
Sep 17 20:57:21.198707 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:21.198627    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5bc11fc5-8747-4c73-aba6-9a4fe0d73ac7-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-pmm4f\" (UID: \"5bc11fc5-8747-4c73-aba6-9a4fe0d73ac7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-pmm4f"
Sep 17 20:57:21.198879 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:21.198724    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw8vs\" (UniqueName: \"kubernetes.io/projected/5bc11fc5-8747-4c73-aba6-9a4fe0d73ac7-kube-api-access-jw8vs\") pod \"cert-manager-operator-controller-manager-684f79c9b9-pmm4f\" (UID: \"5bc11fc5-8747-4c73-aba6-9a4fe0d73ac7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-pmm4f"
Sep 17 20:57:21.299717 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:21.299680    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw8vs\" (UniqueName: \"kubernetes.io/projected/5bc11fc5-8747-4c73-aba6-9a4fe0d73ac7-kube-api-access-jw8vs\") pod \"cert-manager-operator-controller-manager-684f79c9b9-pmm4f\" (UID: \"5bc11fc5-8747-4c73-aba6-9a4fe0d73ac7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-pmm4f"
Sep 17 20:57:21.299926 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:21.299782    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5bc11fc5-8747-4c73-aba6-9a4fe0d73ac7-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-pmm4f\" (UID: \"5bc11fc5-8747-4c73-aba6-9a4fe0d73ac7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-pmm4f"
Sep 17 20:57:21.300160 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:21.300136    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5bc11fc5-8747-4c73-aba6-9a4fe0d73ac7-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-pmm4f\" (UID: \"5bc11fc5-8747-4c73-aba6-9a4fe0d73ac7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-pmm4f"
Sep 17 20:57:21.317603 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:21.317570    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw8vs\" (UniqueName: \"kubernetes.io/projected/5bc11fc5-8747-4c73-aba6-9a4fe0d73ac7-kube-api-access-jw8vs\") pod \"cert-manager-operator-controller-manager-684f79c9b9-pmm4f\" (UID: \"5bc11fc5-8747-4c73-aba6-9a4fe0d73ac7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-pmm4f"
Sep 17 20:57:21.455781 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:21.455741    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-pmm4f"
Sep 17 20:57:21.753047 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:21.752962    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-pmm4f"]
Sep 17 20:57:21.756191 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:57:21.756155    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bc11fc5_8747_4c73_aba6_9a4fe0d73ac7.slice/crio-608d2cd95359aec01a84b7e908cf18803060a7239ade3a3c122475f248efde5b WatchSource:0}: Error finding container 608d2cd95359aec01a84b7e908cf18803060a7239ade3a3c122475f248efde5b: Status 404 returned error can't find the container with id 608d2cd95359aec01a84b7e908cf18803060a7239ade3a3c122475f248efde5b
Sep 17 20:57:22.420940 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:22.420900    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-pmm4f" event={"ID":"5bc11fc5-8747-4c73-aba6-9a4fe0d73ac7","Type":"ContainerStarted","Data":"608d2cd95359aec01a84b7e908cf18803060a7239ade3a3c122475f248efde5b"}
Sep 17 20:57:22.814404 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:22.814366    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:57:22.932256 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:22.932209    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:57:23.046460 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:23.046414    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:57:26.443952 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:26.443913    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-pmm4f" event={"ID":"5bc11fc5-8747-4c73-aba6-9a4fe0d73ac7","Type":"ContainerStarted","Data":"999ad98239a07100e278694caa38c0c4e66e50312af254e087b54c216d032ccd"}
Sep 17 20:57:26.457468 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:26.457437    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:57:26.534481 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:26.534403    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-pmm4f" podStartSLOduration=1.6666948320000001 podStartE2EDuration="5.534383128s" totalImagesPullingTime="3.867688296s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:57:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:57:26.53334466 +0000 UTC m=+511.685789047" watchObservedRunningTime="2026-09-17 20:57:26.534383128 +0000 UTC m=+511.686827505"
Sep 17 20:57:26.688825 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:26.688791    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:57:27.159833 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:27.159799    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-2h479"]
Sep 17 20:57:27.163285 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:27.163266    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-2h479"
Sep 17 20:57:27.165608 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:27.165591    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 17 20:57:27.166243 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:27.166228    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-tlbpb\""
Sep 17 20:57:27.167923 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:27.167908    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 17 20:57:27.186116 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:27.186095    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-2h479"]
Sep 17 20:57:27.252511 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:27.252489    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fa603cd1-aafe-44e4-ad39-384a0e97e912-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-2h479\" (UID: \"fa603cd1-aafe-44e4-ad39-384a0e97e912\") " pod="cert-manager/cert-manager-webhook-6486ffd997-2h479"
Sep 17 20:57:27.252621 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:27.252521    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhpm2\" (UniqueName: \"kubernetes.io/projected/fa603cd1-aafe-44e4-ad39-384a0e97e912-kube-api-access-rhpm2\") pod \"cert-manager-webhook-6486ffd997-2h479\" (UID: \"fa603cd1-aafe-44e4-ad39-384a0e97e912\") " pod="cert-manager/cert-manager-webhook-6486ffd997-2h479"
Sep 17 20:57:27.353620 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:27.353597    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fa603cd1-aafe-44e4-ad39-384a0e97e912-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-2h479\" (UID: \"fa603cd1-aafe-44e4-ad39-384a0e97e912\") " pod="cert-manager/cert-manager-webhook-6486ffd997-2h479"
Sep 17 20:57:27.353718 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:27.353627    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhpm2\" (UniqueName: \"kubernetes.io/projected/fa603cd1-aafe-44e4-ad39-384a0e97e912-kube-api-access-rhpm2\") pod \"cert-manager-webhook-6486ffd997-2h479\" (UID: \"fa603cd1-aafe-44e4-ad39-384a0e97e912\") " pod="cert-manager/cert-manager-webhook-6486ffd997-2h479"
Sep 17 20:57:27.372368 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:27.372343    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhpm2\" (UniqueName: \"kubernetes.io/projected/fa603cd1-aafe-44e4-ad39-384a0e97e912-kube-api-access-rhpm2\") pod \"cert-manager-webhook-6486ffd997-2h479\" (UID: \"fa603cd1-aafe-44e4-ad39-384a0e97e912\") " pod="cert-manager/cert-manager-webhook-6486ffd997-2h479"
Sep 17 20:57:27.372550 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:27.372534    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fa603cd1-aafe-44e4-ad39-384a0e97e912-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-2h479\" (UID: \"fa603cd1-aafe-44e4-ad39-384a0e97e912\") " pod="cert-manager/cert-manager-webhook-6486ffd997-2h479"
Sep 17 20:57:27.482807 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:27.482780    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-2h479"
Sep 17 20:57:27.805189 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:27.805161    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-2h479"]
Sep 17 20:57:27.807646 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:57:27.807616    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa603cd1_aafe_44e4_ad39_384a0e97e912.slice/crio-8e827c3e15f85814b4b18d979006b751e31a005ab902e4aaec092b8ba95fce92 WatchSource:0}: Error finding container 8e827c3e15f85814b4b18d979006b751e31a005ab902e4aaec092b8ba95fce92: Status 404 returned error can't find the container with id 8e827c3e15f85814b4b18d979006b751e31a005ab902e4aaec092b8ba95fce92
Sep 17 20:57:28.453920 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:28.453887    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-2h479" event={"ID":"fa603cd1-aafe-44e4-ad39-384a0e97e912","Type":"ContainerStarted","Data":"8e827c3e15f85814b4b18d979006b751e31a005ab902e4aaec092b8ba95fce92"}
Sep 17 20:57:28.674245 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:28.674209    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:57:28.953001 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:28.952965    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:57:29.279894 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:29.279791    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:57:29.525800 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:29.525766    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:57:32.473488 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:32.473455    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-2h479" event={"ID":"fa603cd1-aafe-44e4-ad39-384a0e97e912","Type":"ContainerStarted","Data":"93f206537fdae81ec57e3990afd47cf7383eef1c1e67df5ef12a50ce68477c5c"}
Sep 17 20:57:32.473895 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:32.473579    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-2h479"
Sep 17 20:57:32.503060 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:32.503016    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-2h479" podStartSLOduration=1.505992645 podStartE2EDuration="5.503002462s" totalImagesPullingTime="3.997009817s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:57:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:57:32.50128839 +0000 UTC m=+517.653732762" watchObservedRunningTime="2026-09-17 20:57:32.503002462 +0000 UTC m=+517.655446837"
Sep 17 20:57:36.350463 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:36.350431    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:57:38.480509 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:38.480471    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-2h479"
Sep 17 20:57:40.242780 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:40.242742    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:57:40.352694 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:40.352652    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:57:42.269791 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:42.269757    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lnggf"]
Sep 17 20:57:42.274665 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:42.274636    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lnggf"
Sep 17 20:57:42.278842 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:42.278820    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 17 20:57:42.279584 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:42.279567    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 17 20:57:42.279690 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:42.279587    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-zqrvb\""
Sep 17 20:57:42.296262 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:42.296233    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lnggf"]
Sep 17 20:57:42.383605 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:42.383575    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bac70aa7-3806-4c7d-9dca-248f98f29612-tmp\") pod \"openshift-lws-operator-fd8ccff4c-lnggf\" (UID: \"bac70aa7-3806-4c7d-9dca-248f98f29612\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lnggf"
Sep 17 20:57:42.383773 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:42.383647    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6h58\" (UniqueName: \"kubernetes.io/projected/bac70aa7-3806-4c7d-9dca-248f98f29612-kube-api-access-j6h58\") pod \"openshift-lws-operator-fd8ccff4c-lnggf\" (UID: \"bac70aa7-3806-4c7d-9dca-248f98f29612\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lnggf"
Sep 17 20:57:42.484543 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:42.484502    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6h58\" (UniqueName: \"kubernetes.io/projected/bac70aa7-3806-4c7d-9dca-248f98f29612-kube-api-access-j6h58\") pod \"openshift-lws-operator-fd8ccff4c-lnggf\" (UID: \"bac70aa7-3806-4c7d-9dca-248f98f29612\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lnggf"
Sep 17 20:57:42.484727 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:42.484585    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bac70aa7-3806-4c7d-9dca-248f98f29612-tmp\") pod \"openshift-lws-operator-fd8ccff4c-lnggf\" (UID: \"bac70aa7-3806-4c7d-9dca-248f98f29612\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lnggf"
Sep 17 20:57:42.485020 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:42.484998    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bac70aa7-3806-4c7d-9dca-248f98f29612-tmp\") pod \"openshift-lws-operator-fd8ccff4c-lnggf\" (UID: \"bac70aa7-3806-4c7d-9dca-248f98f29612\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lnggf"
Sep 17 20:57:42.503552 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:42.503523    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6h58\" (UniqueName: \"kubernetes.io/projected/bac70aa7-3806-4c7d-9dca-248f98f29612-kube-api-access-j6h58\") pod \"openshift-lws-operator-fd8ccff4c-lnggf\" (UID: \"bac70aa7-3806-4c7d-9dca-248f98f29612\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lnggf"
Sep 17 20:57:42.584435 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:42.584380    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lnggf"
Sep 17 20:57:42.874670 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:42.874632    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lnggf"]
Sep 17 20:57:42.877026 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:57:42.876997    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbac70aa7_3806_4c7d_9dca_248f98f29612.slice/crio-0cf0476dd91bd66006689b2aef4f8f48101c44dc496f5038511ce73674ac49f2 WatchSource:0}: Error finding container 0cf0476dd91bd66006689b2aef4f8f48101c44dc496f5038511ce73674ac49f2: Status 404 returned error can't find the container with id 0cf0476dd91bd66006689b2aef4f8f48101c44dc496f5038511ce73674ac49f2
Sep 17 20:57:43.525511 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:43.525451    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lnggf" event={"ID":"bac70aa7-3806-4c7d-9dca-248f98f29612","Type":"ContainerStarted","Data":"0cf0476dd91bd66006689b2aef4f8f48101c44dc496f5038511ce73674ac49f2"}
Sep 17 20:57:45.536713 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:45.536622    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lnggf" event={"ID":"bac70aa7-3806-4c7d-9dca-248f98f29612","Type":"ContainerStarted","Data":"dcc219b599c256a05b8c98c4e8bee691b7c1ec5ef6085fa5dd9e7a77e70a7a48"}
Sep 17 20:57:45.560907 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:45.560860    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lnggf" podStartSLOduration=1.152381711 podStartE2EDuration="3.560831187s" totalImagesPullingTime="2.408449476s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:57:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:57:45.558985338 +0000 UTC m=+530.711429713" watchObservedRunningTime="2026-09-17 20:57:45.560831187 +0000 UTC m=+530.713275563"
Sep 17 20:57:48.634915 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:48.634819    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:57:48.732837 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:48.732805    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:57:52.665393 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:52.665340    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:57:57.775921 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:57.775887    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:57:57.838568 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:57.838533    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:57:58.911879 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:57:58.911834    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:58:02.137085 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:02.137050    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:58:02.250359 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:02.250322    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:58:06.230291 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:06.230247    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-92k25"]
Sep 17 20:58:06.234409 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:06.234385    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-92k25"
Sep 17 20:58:06.237715 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:06.237694    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-mft2s\""
Sep 17 20:58:06.237941 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:06.237914    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 17 20:58:06.250373 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:06.250346    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-92k25"]
Sep 17 20:58:06.272182 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:06.272153    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc057dcf-d949-4624-b5c7-631b60bbc740-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-92k25\" (UID: \"bc057dcf-d949-4624-b5c7-631b60bbc740\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-92k25"
Sep 17 20:58:06.272348 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:06.272191    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc057dcf-d949-4624-b5c7-631b60bbc740-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-92k25\" (UID: \"bc057dcf-d949-4624-b5c7-631b60bbc740\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-92k25"
Sep 17 20:58:06.272348 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:06.272214    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc057dcf-d949-4624-b5c7-631b60bbc740-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-92k25\" (UID: \"bc057dcf-d949-4624-b5c7-631b60bbc740\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-92k25"
Sep 17 20:58:06.272348 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:06.272241    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc057dcf-d949-4624-b5c7-631b60bbc740-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-92k25\" (UID: \"bc057dcf-d949-4624-b5c7-631b60bbc740\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-92k25"
Sep 17 20:58:06.272348 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:06.272309    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc057dcf-d949-4624-b5c7-631b60bbc740-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-92k25\" (UID: \"bc057dcf-d949-4624-b5c7-631b60bbc740\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-92k25"
Sep 17 20:58:06.272497 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:06.272359    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc057dcf-d949-4624-b5c7-631b60bbc740-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-92k25\" (UID: \"bc057dcf-d949-4624-b5c7-631b60bbc740\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-92k25"
Sep 17 20:58:06.272497 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:06.272375    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc057dcf-d949-4624-b5c7-631b60bbc740-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-92k25\" (UID: \"bc057dcf-d949-4624-b5c7-631b60bbc740\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-92k25"
Sep 17 20:58:06.272497 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:06.272422    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frkzq\" (UniqueName: \"kubernetes.io/projected/bc057dcf-d949-4624-b5c7-631b60bbc740-kube-api-access-frkzq\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-92k25\" (UID: \"bc057dcf-d949-4624-b5c7-631b60bbc740\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-92k25"
Sep 17 20:58:06.272497 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:06.272451    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc057dcf-d949-4624-b5c7-631b60bbc740-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-92k25\" (UID: \"bc057dcf-d949-4624-b5c7-631b60bbc740\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-92k25"
Sep 17 20:58:06.284727 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:06.284700    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:58:06.351630 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:06.351589    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:58:06.373624 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:06.373589    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frkzq\" (UniqueName: \"kubernetes.io/projected/bc057dcf-d949-4624-b5c7-631b60bbc740-kube-api-access-frkzq\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-92k25\" (UID: \"bc057dcf-d949-4624-b5c7-631b60bbc740\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-92k25"
Sep 17 20:58:06.373788 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:06.373644    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc057dcf-d949-4624-b5c7-631b60bbc740-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-92k25\" (UID: \"bc057dcf-d949-4624-b5c7-631b60bbc740\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-92k25"
Sep 17 20:58:06.373788 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:06.373705    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc057dcf-d949-4624-b5c7-631b60bbc740-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-92k25\" (UID: \"bc057dcf-d949-4624-b5c7-631b60bbc740\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-92k25"
Sep 17 20:58:06.373936 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:06.373832    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc057dcf-d949-4624-b5c7-631b60bbc740-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-92k25\" (UID: \"bc057dcf-d949-4624-b5c7-631b60bbc740\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-92k25"
Sep 17 20:58:06.373936 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:06.373911    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc057dcf-d949-4624-b5c7-631b60bbc740-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-92k25\" (UID: \"bc057dcf-d949-4624-b5c7-631b60bbc740\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-92k25"
Sep 17 20:58:06.374033 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:06.373959    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc057dcf-d949-4624-b5c7-631b60bbc740-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-92k25\" (UID: \"bc057dcf-d949-4624-b5c7-631b60bbc740\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-92k25"
Sep 17 20:58:06.374138 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:06.374112    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc057dcf-d949-4624-b5c7-631b60bbc740-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-92k25\" (UID: \"bc057dcf-d949-4624-b5c7-631b60bbc740\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-92k25"
Sep 17 20:58:06.374226 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:06.374174    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc057dcf-d949-4624-b5c7-631b60bbc740-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-92k25\" (UID: \"bc057dcf-d949-4624-b5c7-631b60bbc740\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-92k25"
Sep 17 20:58:06.374226 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:06.374199    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc057dcf-d949-4624-b5c7-631b60bbc740-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-92k25\" (UID: \"bc057dcf-d949-4624-b5c7-631b60bbc740\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-92k25"
Sep 17 20:58:06.374332 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:06.374289    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc057dcf-d949-4624-b5c7-631b60bbc740-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-92k25\" (UID: \"bc057dcf-d949-4624-b5c7-631b60bbc740\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-92k25"
Sep 17 20:58:06.374389 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:06.374337    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc057dcf-d949-4624-b5c7-631b60bbc740-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-92k25\" (UID: \"bc057dcf-d949-4624-b5c7-631b60bbc740\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-92k25"
Sep 17 20:58:06.374519 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:06.374499    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc057dcf-d949-4624-b5c7-631b60bbc740-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-92k25\" (UID: \"bc057dcf-d949-4624-b5c7-631b60bbc740\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-92k25"
Sep 17 20:58:06.374563 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:06.374527    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc057dcf-d949-4624-b5c7-631b60bbc740-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-92k25\" (UID: \"bc057dcf-d949-4624-b5c7-631b60bbc740\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-92k25"
Sep 17 20:58:06.374799 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:06.374782    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc057dcf-d949-4624-b5c7-631b60bbc740-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-92k25\" (UID: \"bc057dcf-d949-4624-b5c7-631b60bbc740\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-92k25"
Sep 17 20:58:06.376063 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:06.376047    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc057dcf-d949-4624-b5c7-631b60bbc740-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-92k25\" (UID: \"bc057dcf-d949-4624-b5c7-631b60bbc740\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-92k25"
Sep 17 20:58:06.376166 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:06.376149    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc057dcf-d949-4624-b5c7-631b60bbc740-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-92k25\" (UID: \"bc057dcf-d949-4624-b5c7-631b60bbc740\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-92k25"
Sep 17 20:58:06.382604 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:06.382575    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc057dcf-d949-4624-b5c7-631b60bbc740-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-92k25\" (UID: \"bc057dcf-d949-4624-b5c7-631b60bbc740\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-92k25"
Sep 17 20:58:06.382754 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:06.382736    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frkzq\" (UniqueName: \"kubernetes.io/projected/bc057dcf-d949-4624-b5c7-631b60bbc740-kube-api-access-frkzq\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-92k25\" (UID: \"bc057dcf-d949-4624-b5c7-631b60bbc740\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-92k25"
Sep 17 20:58:06.545751 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:06.545665    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-92k25"
Sep 17 20:58:06.821427 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:06.821395    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-92k25"]
Sep 17 20:58:06.823577 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:58:06.823552    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc057dcf_d949_4624_b5c7_631b60bbc740.slice/crio-d0828871bf57cafee68362d2bdec9dded23e6979fd163a910b845e1e4d10e5e0 WatchSource:0}: Error finding container d0828871bf57cafee68362d2bdec9dded23e6979fd163a910b845e1e4d10e5e0: Status 404 returned error can't find the container with id d0828871bf57cafee68362d2bdec9dded23e6979fd163a910b845e1e4d10e5e0
Sep 17 20:58:07.647828 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:07.647790    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-92k25" event={"ID":"bc057dcf-d949-4624-b5c7-631b60bbc740","Type":"ContainerStarted","Data":"d0828871bf57cafee68362d2bdec9dded23e6979fd163a910b845e1e4d10e5e0"}
Sep 17 20:58:09.413876 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:09.413790    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:58:09.667307 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:09.667270    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 20:58:09.667390 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:09.667366    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 20:58:09.667427 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:09.667395    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 20:58:10.662595 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:10.662557    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-92k25" event={"ID":"bc057dcf-d949-4624-b5c7-631b60bbc740","Type":"ContainerStarted","Data":"4b7fe4a3031135f71716fc8ff25e9183762b05fbed00f36806f765ef4ed39c36"}
Sep 17 20:58:10.685604 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:10.685558    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-92k25" podStartSLOduration=1.8441268709999998 podStartE2EDuration="4.685541224s" totalImagesPullingTime="2.841414353s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:58:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:58:10.683370684 +0000 UTC m=+555.835815061" watchObservedRunningTime="2026-09-17 20:58:10.685541224 +0000 UTC m=+555.837985601"
Sep 17 20:58:11.546885 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:11.546827    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-92k25"
Sep 17 20:58:11.551709 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:11.551683    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-92k25"
Sep 17 20:58:11.666823 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:11.666790    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-92k25"
Sep 17 20:58:11.667958 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:11.667941    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-92k25"
Sep 17 20:58:13.782760 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:13.782727    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:58:13.886306 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:13.886265    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:58:16.640704 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:16.640665    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:58:16.713296 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:16.713264    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7dbd69c77d-67kg9"]
Sep 17 20:58:16.718992 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:16.718969    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7dbd69c77d-67kg9"
Sep 17 20:58:16.722582 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:16.722558    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 17 20:58:16.722582 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:16.722558    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-nkhp9\""
Sep 17 20:58:16.723086 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:16.723066    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 17 20:58:16.723225 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:16.723067    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 17 20:58:16.735926 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:16.735901    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7dbd69c77d-67kg9"]
Sep 17 20:58:16.736827 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:16.736793    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:58:16.770837 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:16.770806    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/48cebd9b-ef8d-46d5-8973-5d5c7a7e83fb-cert\") pod \"lws-controller-manager-7dbd69c77d-67kg9\" (UID: \"48cebd9b-ef8d-46d5-8973-5d5c7a7e83fb\") " pod="openshift-lws-operator/lws-controller-manager-7dbd69c77d-67kg9"
Sep 17 20:58:16.771020 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:16.770911    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7rmv\" (UniqueName: \"kubernetes.io/projected/48cebd9b-ef8d-46d5-8973-5d5c7a7e83fb-kube-api-access-q7rmv\") pod \"lws-controller-manager-7dbd69c77d-67kg9\" (UID: \"48cebd9b-ef8d-46d5-8973-5d5c7a7e83fb\") " pod="openshift-lws-operator/lws-controller-manager-7dbd69c77d-67kg9"
Sep 17 20:58:16.771020 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:16.770974    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/48cebd9b-ef8d-46d5-8973-5d5c7a7e83fb-manager-config\") pod \"lws-controller-manager-7dbd69c77d-67kg9\" (UID: \"48cebd9b-ef8d-46d5-8973-5d5c7a7e83fb\") " pod="openshift-lws-operator/lws-controller-manager-7dbd69c77d-67kg9"
Sep 17 20:58:16.771020 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:16.771015    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/48cebd9b-ef8d-46d5-8973-5d5c7a7e83fb-metrics-cert\") pod \"lws-controller-manager-7dbd69c77d-67kg9\" (UID: \"48cebd9b-ef8d-46d5-8973-5d5c7a7e83fb\") " pod="openshift-lws-operator/lws-controller-manager-7dbd69c77d-67kg9"
Sep 17 20:58:16.871674 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:16.871642    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/48cebd9b-ef8d-46d5-8973-5d5c7a7e83fb-manager-config\") pod \"lws-controller-manager-7dbd69c77d-67kg9\" (UID: \"48cebd9b-ef8d-46d5-8973-5d5c7a7e83fb\") " pod="openshift-lws-operator/lws-controller-manager-7dbd69c77d-67kg9"
Sep 17 20:58:16.871674 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:16.871685    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/48cebd9b-ef8d-46d5-8973-5d5c7a7e83fb-metrics-cert\") pod \"lws-controller-manager-7dbd69c77d-67kg9\" (UID: \"48cebd9b-ef8d-46d5-8973-5d5c7a7e83fb\") " pod="openshift-lws-operator/lws-controller-manager-7dbd69c77d-67kg9"
Sep 17 20:58:16.871967 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:16.871724    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/48cebd9b-ef8d-46d5-8973-5d5c7a7e83fb-cert\") pod \"lws-controller-manager-7dbd69c77d-67kg9\" (UID: \"48cebd9b-ef8d-46d5-8973-5d5c7a7e83fb\") " pod="openshift-lws-operator/lws-controller-manager-7dbd69c77d-67kg9"
Sep 17 20:58:16.871967 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:16.871791    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7rmv\" (UniqueName: \"kubernetes.io/projected/48cebd9b-ef8d-46d5-8973-5d5c7a7e83fb-kube-api-access-q7rmv\") pod \"lws-controller-manager-7dbd69c77d-67kg9\" (UID: \"48cebd9b-ef8d-46d5-8973-5d5c7a7e83fb\") " pod="openshift-lws-operator/lws-controller-manager-7dbd69c77d-67kg9"
Sep 17 20:58:16.872430 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:16.872405    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/48cebd9b-ef8d-46d5-8973-5d5c7a7e83fb-manager-config\") pod \"lws-controller-manager-7dbd69c77d-67kg9\" (UID: \"48cebd9b-ef8d-46d5-8973-5d5c7a7e83fb\") " pod="openshift-lws-operator/lws-controller-manager-7dbd69c77d-67kg9"
Sep 17 20:58:16.874129 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:16.874101    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/48cebd9b-ef8d-46d5-8973-5d5c7a7e83fb-cert\") pod \"lws-controller-manager-7dbd69c77d-67kg9\" (UID: \"48cebd9b-ef8d-46d5-8973-5d5c7a7e83fb\") " pod="openshift-lws-operator/lws-controller-manager-7dbd69c77d-67kg9"
Sep 17 20:58:16.874225 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:16.874189    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/48cebd9b-ef8d-46d5-8973-5d5c7a7e83fb-metrics-cert\") pod \"lws-controller-manager-7dbd69c77d-67kg9\" (UID: \"48cebd9b-ef8d-46d5-8973-5d5c7a7e83fb\") " pod="openshift-lws-operator/lws-controller-manager-7dbd69c77d-67kg9"
Sep 17 20:58:16.888747 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:16.888722    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7rmv\" (UniqueName: \"kubernetes.io/projected/48cebd9b-ef8d-46d5-8973-5d5c7a7e83fb-kube-api-access-q7rmv\") pod \"lws-controller-manager-7dbd69c77d-67kg9\" (UID: \"48cebd9b-ef8d-46d5-8973-5d5c7a7e83fb\") " pod="openshift-lws-operator/lws-controller-manager-7dbd69c77d-67kg9"
Sep 17 20:58:17.029235 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:17.029201    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7dbd69c77d-67kg9"
Sep 17 20:58:17.338550 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:17.338520    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7dbd69c77d-67kg9"]
Sep 17 20:58:17.339260 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:58:17.339233    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48cebd9b_ef8d_46d5_8973_5d5c7a7e83fb.slice/crio-728d397f608105cd4e80959187ec34eb97faa4ec0ec9fb1605f1a47388cc81d8 WatchSource:0}: Error finding container 728d397f608105cd4e80959187ec34eb97faa4ec0ec9fb1605f1a47388cc81d8: Status 404 returned error can't find the container with id 728d397f608105cd4e80959187ec34eb97faa4ec0ec9fb1605f1a47388cc81d8
Sep 17 20:58:17.694979 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:17.694873    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7dbd69c77d-67kg9" event={"ID":"48cebd9b-ef8d-46d5-8973-5d5c7a7e83fb","Type":"ContainerStarted","Data":"728d397f608105cd4e80959187ec34eb97faa4ec0ec9fb1605f1a47388cc81d8"}
Sep 17 20:58:19.707912 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:19.707873    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7dbd69c77d-67kg9" event={"ID":"48cebd9b-ef8d-46d5-8973-5d5c7a7e83fb","Type":"ContainerStarted","Data":"9f0a4362748af7e63a1e9eb85b59f9f379fdd6beb437e6ced6e464ebf3f3fe08"}
Sep 17 20:58:19.708352 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:19.708048    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7dbd69c77d-67kg9"
Sep 17 20:58:19.745594 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:19.745533    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7dbd69c77d-67kg9" podStartSLOduration=2.151589395 podStartE2EDuration="3.745513843s" totalImagesPullingTime="1.593924448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:58:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:58:19.744343495 +0000 UTC m=+564.896787873" watchObservedRunningTime="2026-09-17 20:58:19.745513843 +0000 UTC m=+564.897958219"
Sep 17 20:58:30.714760 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:30.714731    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7dbd69c77d-67kg9"
Sep 17 20:58:34.751211 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:34.751164    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:58:35.554979 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:35.554940    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:58:38.630550 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:38.630512    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:58:38.769256 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:38.769217    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:58:40.684172 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:40.684129    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:58:40.781538 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:40.781498    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:58:41.346015 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:41.345972    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:58:41.627926 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:41.627822    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:58:41.725274 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:41.725230    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:58:41.849620 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:41.849582    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:58:45.161102 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:45.161056    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:58:45.825092 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:45.825062    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:58:48.394415 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:48.394372    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:58:48.497362 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:48.497324    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:58:48.785576 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:48.785547    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:58:48.897520 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:48.897491    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:58:51.871353 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:51.871322    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:58:52.043573 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:52.043538    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:58:52.574559 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:52.574525    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:58:52.762633 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:52.762596    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:58:56.345279 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:56.345241    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:58:56.451237 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:56.451192    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:58:56.942746 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:56.942714    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:58:57.040904 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:57.040867    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:58:57.141525 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:57.141491    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:58:59.842331 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:58:59.842297    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:59:03.424421 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:03.424377    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:59:03.479227 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:03.479179    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:59:09.000862 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:09.000827    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:59:09.095503 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:09.095467    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:59:09.576250 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:09.576213    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:59:09.635838 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:09.635802    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:59:12.023154 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:12.023121    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:59:12.119454 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:12.119407    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:59:17.192360 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:17.192315    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:59:17.287312 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:17.287271    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:59:17.419128 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:17.419098    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:59:17.519475 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:17.519429    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:59:17.621483 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:17.621446    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:59:17.720807 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:17.720774    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:59:23.259776 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:23.259743    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:59:23.359369 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:23.359332    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:59:36.691777 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:36.691720    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:59:36.791727 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:36.791694    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:59:36.891173 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:36.891136    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:59:37.496271 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:37.496240    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-h8bvk"]
Sep 17 20:59:37.499685 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:37.499663    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-h8bvk"
Sep 17 20:59:37.502052 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:37.502025    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 17 20:59:37.502151 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:37.502071    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 17 20:59:37.502221 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:37.502204    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-4dfl6\""
Sep 17 20:59:37.509216 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:37.509196    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-h8bvk"]
Sep 17 20:59:37.543899 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:37.543834    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:59:37.631152 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:37.631125    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-jzsjd"]
Sep 17 20:59:37.633404 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:37.633382    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:59:37.634404 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:37.634388    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-jzsjd"
Sep 17 20:59:37.642997 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:37.642975    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-jzsjd"]
Sep 17 20:59:37.685025 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:37.684999    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcm4v\" (UniqueName: \"kubernetes.io/projected/95d52cf0-6d86-4304-ad79-e5d60be7ac83-kube-api-access-gcm4v\") pod \"authorino-7bcfd8f9cd-h8bvk\" (UID: \"95d52cf0-6d86-4304-ad79-e5d60be7ac83\") " pod="kuadrant-system/authorino-7bcfd8f9cd-h8bvk"
Sep 17 20:59:37.786053 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:37.785979    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hqlv\" (UniqueName: \"kubernetes.io/projected/ac93b896-7bfe-4b51-9fba-fdf3dd46f7da-kube-api-access-5hqlv\") pod \"authorino-59c974bd48-jzsjd\" (UID: \"ac93b896-7bfe-4b51-9fba-fdf3dd46f7da\") " pod="kuadrant-system/authorino-59c974bd48-jzsjd"
Sep 17 20:59:37.786383 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:37.786081    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcm4v\" (UniqueName: \"kubernetes.io/projected/95d52cf0-6d86-4304-ad79-e5d60be7ac83-kube-api-access-gcm4v\") pod \"authorino-7bcfd8f9cd-h8bvk\" (UID: \"95d52cf0-6d86-4304-ad79-e5d60be7ac83\") " pod="kuadrant-system/authorino-7bcfd8f9cd-h8bvk"
Sep 17 20:59:37.794560 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:37.794532    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcm4v\" (UniqueName: \"kubernetes.io/projected/95d52cf0-6d86-4304-ad79-e5d60be7ac83-kube-api-access-gcm4v\") pod \"authorino-7bcfd8f9cd-h8bvk\" (UID: \"95d52cf0-6d86-4304-ad79-e5d60be7ac83\") " pod="kuadrant-system/authorino-7bcfd8f9cd-h8bvk"
Sep 17 20:59:37.809319 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:37.809300    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-h8bvk"
Sep 17 20:59:37.886914 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:37.886874    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hqlv\" (UniqueName: \"kubernetes.io/projected/ac93b896-7bfe-4b51-9fba-fdf3dd46f7da-kube-api-access-5hqlv\") pod \"authorino-59c974bd48-jzsjd\" (UID: \"ac93b896-7bfe-4b51-9fba-fdf3dd46f7da\") " pod="kuadrant-system/authorino-59c974bd48-jzsjd"
Sep 17 20:59:37.895560 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:37.895536    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hqlv\" (UniqueName: \"kubernetes.io/projected/ac93b896-7bfe-4b51-9fba-fdf3dd46f7da-kube-api-access-5hqlv\") pod \"authorino-59c974bd48-jzsjd\" (UID: \"ac93b896-7bfe-4b51-9fba-fdf3dd46f7da\") " pod="kuadrant-system/authorino-59c974bd48-jzsjd"
Sep 17 20:59:37.943423 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:37.943398    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-jzsjd"
Sep 17 20:59:38.077064 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:38.076970    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-h8bvk"]
Sep 17 20:59:38.079742 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:59:38.079718    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod95d52cf0_6d86_4304_ad79_e5d60be7ac83.slice/crio-341b24f2bcb85519f9d3e60c7cae704215ba94fa0d34a82143befc28fe3054d8 WatchSource:0}: Error finding container 341b24f2bcb85519f9d3e60c7cae704215ba94fa0d34a82143befc28fe3054d8: Status 404 returned error can't find the container with id 341b24f2bcb85519f9d3e60c7cae704215ba94fa0d34a82143befc28fe3054d8
Sep 17 20:59:38.211380 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:38.211347    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-jzsjd"]
Sep 17 20:59:38.213829 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:59:38.213806    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podac93b896_7bfe_4b51_9fba_fdf3dd46f7da.slice/crio-a7a0610b643a6d2300cd1daa7d75dce29611d1d0e690fd30883d009020ea9ccb WatchSource:0}: Error finding container a7a0610b643a6d2300cd1daa7d75dce29611d1d0e690fd30883d009020ea9ccb: Status 404 returned error can't find the container with id a7a0610b643a6d2300cd1daa7d75dce29611d1d0e690fd30883d009020ea9ccb
Sep 17 20:59:39.077331 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:39.077290    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-h8bvk" event={"ID":"95d52cf0-6d86-4304-ad79-e5d60be7ac83","Type":"ContainerStarted","Data":"341b24f2bcb85519f9d3e60c7cae704215ba94fa0d34a82143befc28fe3054d8"}
Sep 17 20:59:39.079009 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:39.078972    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-jzsjd" event={"ID":"ac93b896-7bfe-4b51-9fba-fdf3dd46f7da","Type":"ContainerStarted","Data":"a7a0610b643a6d2300cd1daa7d75dce29611d1d0e690fd30883d009020ea9ccb"}
Sep 17 20:59:39.832684 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:39.832651    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:59:40.085940 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:40.085838    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-jzsjd" event={"ID":"ac93b896-7bfe-4b51-9fba-fdf3dd46f7da","Type":"ContainerStarted","Data":"b315ff1b33905d53f0fd62d8ae3f4bd8da9be958081a6bec7811945b1df46439"}
Sep 17 20:59:40.087456 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:40.087431    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-h8bvk" event={"ID":"95d52cf0-6d86-4304-ad79-e5d60be7ac83","Type":"ContainerStarted","Data":"ff4b614ee8505a2d9325fc1e1a23f14f3384136ebb4fd9726b823d3f83643725"}
Sep 17 20:59:40.101430 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:40.101383    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-jzsjd" podStartSLOduration=1.522571611 podStartE2EDuration="3.101369631s" totalImagesPullingTime="1.57879802s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:59:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:59:40.099772249 +0000 UTC m=+645.252216627" watchObservedRunningTime="2026-09-17 20:59:40.101369631 +0000 UTC m=+645.253814060"
Sep 17 20:59:40.114318 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:40.114267    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-h8bvk" podStartSLOduration=1.397305819 podStartE2EDuration="3.114251644s" totalImagesPullingTime="1.716945825s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:59:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:59:40.113281764 +0000 UTC m=+645.265726141" watchObservedRunningTime="2026-09-17 20:59:40.114251644 +0000 UTC m=+645.266696022"
Sep 17 20:59:40.140071 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:40.140039    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-h8bvk"]
Sep 17 20:59:42.096321 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:42.096282    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-h8bvk" podUID="95d52cf0-6d86-4304-ad79-e5d60be7ac83" containerName="authorino" containerID="cri-o://ff4b614ee8505a2d9325fc1e1a23f14f3384136ebb4fd9726b823d3f83643725" gracePeriod=30
Sep 17 20:59:42.344391 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:42.344363    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-h8bvk"
Sep 17 20:59:42.423905 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:42.423825    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gcm4v\" (UniqueName: \"kubernetes.io/projected/95d52cf0-6d86-4304-ad79-e5d60be7ac83-kube-api-access-gcm4v\") pod \"95d52cf0-6d86-4304-ad79-e5d60be7ac83\" (UID: \"95d52cf0-6d86-4304-ad79-e5d60be7ac83\") "
Sep 17 20:59:42.426021 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:42.425994    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95d52cf0-6d86-4304-ad79-e5d60be7ac83-kube-api-access-gcm4v" (OuterVolumeSpecName: "kube-api-access-gcm4v") pod "95d52cf0-6d86-4304-ad79-e5d60be7ac83" (UID: "95d52cf0-6d86-4304-ad79-e5d60be7ac83"). InnerVolumeSpecName "kube-api-access-gcm4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:59:42.524651 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:42.524625    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcm4v\" (UniqueName: \"kubernetes.io/projected/95d52cf0-6d86-4304-ad79-e5d60be7ac83-kube-api-access-gcm4v\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 20:59:43.101809 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:43.101772    2575 generic.go:356] "Generic (PLEG): container finished" podID="95d52cf0-6d86-4304-ad79-e5d60be7ac83" containerID="ff4b614ee8505a2d9325fc1e1a23f14f3384136ebb4fd9726b823d3f83643725" exitCode=0
Sep 17 20:59:43.102220 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:43.101817    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-h8bvk"
Sep 17 20:59:43.102220 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:43.101877    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-h8bvk" event={"ID":"95d52cf0-6d86-4304-ad79-e5d60be7ac83","Type":"ContainerDied","Data":"ff4b614ee8505a2d9325fc1e1a23f14f3384136ebb4fd9726b823d3f83643725"}
Sep 17 20:59:43.102220 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:43.101916    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-h8bvk" event={"ID":"95d52cf0-6d86-4304-ad79-e5d60be7ac83","Type":"ContainerDied","Data":"341b24f2bcb85519f9d3e60c7cae704215ba94fa0d34a82143befc28fe3054d8"}
Sep 17 20:59:43.102220 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:43.101934    2575 scope.go:117] "RemoveContainer" containerID="ff4b614ee8505a2d9325fc1e1a23f14f3384136ebb4fd9726b823d3f83643725"
Sep 17 20:59:43.112908 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:43.112891    2575 scope.go:117] "RemoveContainer" containerID="ff4b614ee8505a2d9325fc1e1a23f14f3384136ebb4fd9726b823d3f83643725"
Sep 17 20:59:43.113204 ip-10-0-139-36 kubenswrapper[2575]: E0917 20:59:43.113186    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff4b614ee8505a2d9325fc1e1a23f14f3384136ebb4fd9726b823d3f83643725\": container with ID starting with ff4b614ee8505a2d9325fc1e1a23f14f3384136ebb4fd9726b823d3f83643725 not found: ID does not exist" containerID="ff4b614ee8505a2d9325fc1e1a23f14f3384136ebb4fd9726b823d3f83643725"
Sep 17 20:59:43.113258 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:43.113213    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff4b614ee8505a2d9325fc1e1a23f14f3384136ebb4fd9726b823d3f83643725"} err="failed to get container status \"ff4b614ee8505a2d9325fc1e1a23f14f3384136ebb4fd9726b823d3f83643725\": rpc error: code = NotFound desc = could not find container \"ff4b614ee8505a2d9325fc1e1a23f14f3384136ebb4fd9726b823d3f83643725\": container with ID starting with ff4b614ee8505a2d9325fc1e1a23f14f3384136ebb4fd9726b823d3f83643725 not found: ID does not exist"
Sep 17 20:59:43.124380 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:43.124358    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-h8bvk"]
Sep 17 20:59:43.127659 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:43.127633    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-h8bvk"]
Sep 17 20:59:43.356432 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:43.356344    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95d52cf0-6d86-4304-ad79-e5d60be7ac83" path="/var/lib/kubelet/pods/95d52cf0-6d86-4304-ad79-e5d60be7ac83/volumes"
Sep 17 20:59:56.244470 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:56.244428    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:59:56.592536 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:56.592446    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 20:59:58.862778 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:58.862744    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d9898bd99-fw2sk"]
Sep 17 20:59:58.863145 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:58.863132    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95d52cf0-6d86-4304-ad79-e5d60be7ac83" containerName="authorino"
Sep 17 20:59:58.863145 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:58.863144    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="95d52cf0-6d86-4304-ad79-e5d60be7ac83" containerName="authorino"
Sep 17 20:59:58.863235 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:58.863223    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="95d52cf0-6d86-4304-ad79-e5d60be7ac83" containerName="authorino"
Sep 17 20:59:58.867343 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:58.867326    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d9898bd99-fw2sk"
Sep 17 20:59:58.870141 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:58.870125    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 17 20:59:58.872916 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:58.872889    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d9898bd99-fw2sk"]
Sep 17 20:59:58.939270 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:58.939242    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3863cd4c-03d9-4b97-ad6f-73b0cd07cb3f-tls-cert\") pod \"authorino-6d9898bd99-fw2sk\" (UID: \"3863cd4c-03d9-4b97-ad6f-73b0cd07cb3f\") " pod="kuadrant-system/authorino-6d9898bd99-fw2sk"
Sep 17 20:59:58.939412 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:58.939280    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn4d8\" (UniqueName: \"kubernetes.io/projected/3863cd4c-03d9-4b97-ad6f-73b0cd07cb3f-kube-api-access-wn4d8\") pod \"authorino-6d9898bd99-fw2sk\" (UID: \"3863cd4c-03d9-4b97-ad6f-73b0cd07cb3f\") " pod="kuadrant-system/authorino-6d9898bd99-fw2sk"
Sep 17 20:59:59.040326 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:59.040300    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3863cd4c-03d9-4b97-ad6f-73b0cd07cb3f-tls-cert\") pod \"authorino-6d9898bd99-fw2sk\" (UID: \"3863cd4c-03d9-4b97-ad6f-73b0cd07cb3f\") " pod="kuadrant-system/authorino-6d9898bd99-fw2sk"
Sep 17 20:59:59.040443 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:59.040332    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn4d8\" (UniqueName: \"kubernetes.io/projected/3863cd4c-03d9-4b97-ad6f-73b0cd07cb3f-kube-api-access-wn4d8\") pod \"authorino-6d9898bd99-fw2sk\" (UID: \"3863cd4c-03d9-4b97-ad6f-73b0cd07cb3f\") " pod="kuadrant-system/authorino-6d9898bd99-fw2sk"
Sep 17 20:59:59.042648 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:59.042625    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3863cd4c-03d9-4b97-ad6f-73b0cd07cb3f-tls-cert\") pod \"authorino-6d9898bd99-fw2sk\" (UID: \"3863cd4c-03d9-4b97-ad6f-73b0cd07cb3f\") " pod="kuadrant-system/authorino-6d9898bd99-fw2sk"
Sep 17 20:59:59.048777 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:59.048757    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn4d8\" (UniqueName: \"kubernetes.io/projected/3863cd4c-03d9-4b97-ad6f-73b0cd07cb3f-kube-api-access-wn4d8\") pod \"authorino-6d9898bd99-fw2sk\" (UID: \"3863cd4c-03d9-4b97-ad6f-73b0cd07cb3f\") " pod="kuadrant-system/authorino-6d9898bd99-fw2sk"
Sep 17 20:59:59.177481 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:59.177414    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d9898bd99-fw2sk"
Sep 17 20:59:59.449217 ip-10-0-139-36 kubenswrapper[2575]: I0917 20:59:59.449192    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d9898bd99-fw2sk"]
Sep 17 20:59:59.451806 ip-10-0-139-36 kubenswrapper[2575]: W0917 20:59:59.451773    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3863cd4c_03d9_4b97_ad6f_73b0cd07cb3f.slice/crio-2dc9254f44c639fe2ef7b180091e6c0a38f59eaaa0ad45ea920b0abc85828cec WatchSource:0}: Error finding container 2dc9254f44c639fe2ef7b180091e6c0a38f59eaaa0ad45ea920b0abc85828cec: Status 404 returned error can't find the container with id 2dc9254f44c639fe2ef7b180091e6c0a38f59eaaa0ad45ea920b0abc85828cec
Sep 17 21:00:00.186488 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:00:00.186394    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d9898bd99-fw2sk" event={"ID":"3863cd4c-03d9-4b97-ad6f-73b0cd07cb3f","Type":"ContainerStarted","Data":"efa6be8c59617babcf1e77132278f5be1fcb325aed217e2bfbd984f351e850ad"}
Sep 17 21:00:00.186488 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:00:00.186440    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d9898bd99-fw2sk" event={"ID":"3863cd4c-03d9-4b97-ad6f-73b0cd07cb3f","Type":"ContainerStarted","Data":"2dc9254f44c639fe2ef7b180091e6c0a38f59eaaa0ad45ea920b0abc85828cec"}
Sep 17 21:00:00.228321 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:00:00.228255    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6d9898bd99-fw2sk" podStartSLOduration=1.828170837 podStartE2EDuration="2.22823219s" totalImagesPullingTime="400.061353ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:59:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:00:00.201339053 +0000 UTC m=+665.353783430" watchObservedRunningTime="2026-09-17 21:00:00.22823219 +0000 UTC m=+665.380676566"
Sep 17 21:00:00.230217 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:00:00.230190    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-jzsjd"]
Sep 17 21:00:00.230415 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:00:00.230393    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-jzsjd" podUID="ac93b896-7bfe-4b51-9fba-fdf3dd46f7da" containerName="authorino" containerID="cri-o://b315ff1b33905d53f0fd62d8ae3f4bd8da9be958081a6bec7811945b1df46439" gracePeriod=30
Sep 17 21:00:00.481870 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:00:00.481829    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-jzsjd"
Sep 17 21:00:00.555084 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:00:00.555054    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5hqlv\" (UniqueName: \"kubernetes.io/projected/ac93b896-7bfe-4b51-9fba-fdf3dd46f7da-kube-api-access-5hqlv\") pod \"ac93b896-7bfe-4b51-9fba-fdf3dd46f7da\" (UID: \"ac93b896-7bfe-4b51-9fba-fdf3dd46f7da\") "
Sep 17 21:00:00.557041 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:00:00.557013    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac93b896-7bfe-4b51-9fba-fdf3dd46f7da-kube-api-access-5hqlv" (OuterVolumeSpecName: "kube-api-access-5hqlv") pod "ac93b896-7bfe-4b51-9fba-fdf3dd46f7da" (UID: "ac93b896-7bfe-4b51-9fba-fdf3dd46f7da"). InnerVolumeSpecName "kube-api-access-5hqlv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:00:00.656122 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:00:00.656089    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hqlv\" (UniqueName: \"kubernetes.io/projected/ac93b896-7bfe-4b51-9fba-fdf3dd46f7da-kube-api-access-5hqlv\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:00:01.196603 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:00:01.196570    2575 generic.go:356] "Generic (PLEG): container finished" podID="ac93b896-7bfe-4b51-9fba-fdf3dd46f7da" containerID="b315ff1b33905d53f0fd62d8ae3f4bd8da9be958081a6bec7811945b1df46439" exitCode=0
Sep 17 21:00:01.197053 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:00:01.196612    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-jzsjd"
Sep 17 21:00:01.197053 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:00:01.196659    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-jzsjd" event={"ID":"ac93b896-7bfe-4b51-9fba-fdf3dd46f7da","Type":"ContainerDied","Data":"b315ff1b33905d53f0fd62d8ae3f4bd8da9be958081a6bec7811945b1df46439"}
Sep 17 21:00:01.197053 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:00:01.196696    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-jzsjd" event={"ID":"ac93b896-7bfe-4b51-9fba-fdf3dd46f7da","Type":"ContainerDied","Data":"a7a0610b643a6d2300cd1daa7d75dce29611d1d0e690fd30883d009020ea9ccb"}
Sep 17 21:00:01.197053 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:00:01.196716    2575 scope.go:117] "RemoveContainer" containerID="b315ff1b33905d53f0fd62d8ae3f4bd8da9be958081a6bec7811945b1df46439"
Sep 17 21:00:01.205239 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:00:01.205219    2575 scope.go:117] "RemoveContainer" containerID="b315ff1b33905d53f0fd62d8ae3f4bd8da9be958081a6bec7811945b1df46439"
Sep 17 21:00:01.205576 ip-10-0-139-36 kubenswrapper[2575]: E0917 21:00:01.205556    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b315ff1b33905d53f0fd62d8ae3f4bd8da9be958081a6bec7811945b1df46439\": container with ID starting with b315ff1b33905d53f0fd62d8ae3f4bd8da9be958081a6bec7811945b1df46439 not found: ID does not exist" containerID="b315ff1b33905d53f0fd62d8ae3f4bd8da9be958081a6bec7811945b1df46439"
Sep 17 21:00:01.205623 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:00:01.205586    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b315ff1b33905d53f0fd62d8ae3f4bd8da9be958081a6bec7811945b1df46439"} err="failed to get container status \"b315ff1b33905d53f0fd62d8ae3f4bd8da9be958081a6bec7811945b1df46439\": rpc error: code = NotFound desc = could not find container \"b315ff1b33905d53f0fd62d8ae3f4bd8da9be958081a6bec7811945b1df46439\": container with ID starting with b315ff1b33905d53f0fd62d8ae3f4bd8da9be958081a6bec7811945b1df46439 not found: ID does not exist"
Sep 17 21:00:01.217161 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:00:01.217132    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-jzsjd"]
Sep 17 21:00:01.220325 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:00:01.220300    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-jzsjd"]
Sep 17 21:00:01.357336 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:00:01.357298    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac93b896-7bfe-4b51-9fba-fdf3dd46f7da" path="/var/lib/kubelet/pods/ac93b896-7bfe-4b51-9fba-fdf3dd46f7da/volumes"
Sep 17 21:00:04.341806 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:00:04.341772    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:00:04.446100 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:00:04.446066    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:00:31.118279 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:00:31.118238    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:00:31.214005 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:00:31.213971    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:00:31.328273 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:00:31.328236    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:00:31.426394 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:00:31.426309    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:00:31.792750 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:00:31.792715    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:00:31.849783 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:00:31.849738    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:00:50.810964 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:00:50.810879    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:00:50.928158 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:00:50.928125    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:01:09.502994 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:01:09.502956    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:01:13.034409 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:01:13.034374    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:01:13.449988 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:01:13.449952    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:01:49.796717 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:01:49.796673    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:01:49.867883 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:01:49.867831    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:01:50.049699 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:01:50.049581    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:01:50.157708 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:01:50.157678    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:01:50.255257 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:01:50.255222    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:01:51.044386 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:01:51.044348    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:01:51.142843 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:01:51.142809    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:02:06.586493 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:02:06.586459    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-b9xfq"]
Sep 17 21:02:06.586947 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:02:06.586838    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac93b896-7bfe-4b51-9fba-fdf3dd46f7da" containerName="authorino"
Sep 17 21:02:06.586947 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:02:06.586863    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac93b896-7bfe-4b51-9fba-fdf3dd46f7da" containerName="authorino"
Sep 17 21:02:06.586947 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:02:06.586924    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac93b896-7bfe-4b51-9fba-fdf3dd46f7da" containerName="authorino"
Sep 17 21:02:06.590037 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:02:06.590021    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-b9xfq"
Sep 17 21:02:06.592557 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:02:06.592531    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 17 21:02:06.592690 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:02:06.592536    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 21:02:06.592690 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:02:06.592538    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-7hgg2\""
Sep 17 21:02:06.592690 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:02:06.592572    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 21:02:06.593081 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:02:06.593065    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 17 21:02:06.599391 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:02:06.599371    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-b9xfq"]
Sep 17 21:02:06.691047 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:02:06.691002    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15c055e0-7da5-4863-90d9-c736126957ff-model-cache\") pod \"s3-init-b9xfq\" (UID: \"15c055e0-7da5-4863-90d9-c736126957ff\") " pod="kserve/s3-init-b9xfq"
Sep 17 21:02:06.691235 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:02:06.691072    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzj4l\" (UniqueName: \"kubernetes.io/projected/15c055e0-7da5-4863-90d9-c736126957ff-kube-api-access-gzj4l\") pod \"s3-init-b9xfq\" (UID: \"15c055e0-7da5-4863-90d9-c736126957ff\") " pod="kserve/s3-init-b9xfq"
Sep 17 21:02:06.792276 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:02:06.792238    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzj4l\" (UniqueName: \"kubernetes.io/projected/15c055e0-7da5-4863-90d9-c736126957ff-kube-api-access-gzj4l\") pod \"s3-init-b9xfq\" (UID: \"15c055e0-7da5-4863-90d9-c736126957ff\") " pod="kserve/s3-init-b9xfq"
Sep 17 21:02:06.792479 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:02:06.792372    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15c055e0-7da5-4863-90d9-c736126957ff-model-cache\") pod \"s3-init-b9xfq\" (UID: \"15c055e0-7da5-4863-90d9-c736126957ff\") " pod="kserve/s3-init-b9xfq"
Sep 17 21:02:06.792783 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:02:06.792760    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15c055e0-7da5-4863-90d9-c736126957ff-model-cache\") pod \"s3-init-b9xfq\" (UID: \"15c055e0-7da5-4863-90d9-c736126957ff\") " pod="kserve/s3-init-b9xfq"
Sep 17 21:02:06.800691 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:02:06.800670    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzj4l\" (UniqueName: \"kubernetes.io/projected/15c055e0-7da5-4863-90d9-c736126957ff-kube-api-access-gzj4l\") pod \"s3-init-b9xfq\" (UID: \"15c055e0-7da5-4863-90d9-c736126957ff\") " pod="kserve/s3-init-b9xfq"
Sep 17 21:02:06.899565 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:02:06.899476    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-b9xfq"
Sep 17 21:02:07.173106 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:02:07.173077    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-b9xfq"]
Sep 17 21:02:07.175469 ip-10-0-139-36 kubenswrapper[2575]: W0917 21:02:07.175437    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod15c055e0_7da5_4863_90d9_c736126957ff.slice/crio-987d83a288344e2a0b3fa603edd035e481cb033338463b2a9ec88757ec12fe09 WatchSource:0}: Error finding container 987d83a288344e2a0b3fa603edd035e481cb033338463b2a9ec88757ec12fe09: Status 404 returned error can't find the container with id 987d83a288344e2a0b3fa603edd035e481cb033338463b2a9ec88757ec12fe09
Sep 17 21:02:07.177474 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:02:07.177455    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 21:02:07.802532 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:02:07.802487    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-b9xfq" event={"ID":"15c055e0-7da5-4863-90d9-c736126957ff","Type":"ContainerStarted","Data":"987d83a288344e2a0b3fa603edd035e481cb033338463b2a9ec88757ec12fe09"}
Sep 17 21:02:10.821037 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:02:10.820991    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-b9xfq" event={"ID":"15c055e0-7da5-4863-90d9-c736126957ff","Type":"ContainerStarted","Data":"d7f2860c353639f4769886655ba204f51dd6a2562531d31e03aa71ab1137d772"}
Sep 17 21:02:12.833624 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:02:12.833588    2575 generic.go:356] "Generic (PLEG): container finished" podID="15c055e0-7da5-4863-90d9-c736126957ff" containerID="d7f2860c353639f4769886655ba204f51dd6a2562531d31e03aa71ab1137d772" exitCode=0
Sep 17 21:02:12.834065 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:02:12.833664    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-b9xfq" event={"ID":"15c055e0-7da5-4863-90d9-c736126957ff","Type":"ContainerDied","Data":"d7f2860c353639f4769886655ba204f51dd6a2562531d31e03aa71ab1137d772"}
Sep 17 21:02:17.867900 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:02:17.867802    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-b9xfq" event={"ID":"15c055e0-7da5-4863-90d9-c736126957ff","Type":"ContainerStarted","Data":"cc5b542192e310523ef95c3b7fc4dc4a01202515717068fc6a8b333bc5478cbf"}
Sep 17 21:02:17.884690 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:02:17.884639    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-b9xfq" podStartSLOduration=2.359594725 podStartE2EDuration="11.884619885s" totalImagesPullingTime="7.933341576s" totalInitContainerRuntime="1.591683584s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:02:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:02:17.883365254 +0000 UTC m=+803.035809633" watchObservedRunningTime="2026-09-17 21:02:17.884619885 +0000 UTC m=+803.037064263"
Sep 17 21:03:12.541706 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:03:12.541670    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:03:12.643377 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:03:12.643343    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:03:23.181491 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:03:23.181455    2575 generic.go:356] "Generic (PLEG): container finished" podID="15c055e0-7da5-4863-90d9-c736126957ff" containerID="cc5b542192e310523ef95c3b7fc4dc4a01202515717068fc6a8b333bc5478cbf" exitCode=0
Sep 17 21:03:23.181930 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:03:23.181528    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-b9xfq" event={"ID":"15c055e0-7da5-4863-90d9-c736126957ff","Type":"ContainerDied","Data":"cc5b542192e310523ef95c3b7fc4dc4a01202515717068fc6a8b333bc5478cbf"}
Sep 17 21:03:24.316796 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:03:24.316769    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-b9xfq"
Sep 17 21:03:24.470720 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:03:24.470692    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15c055e0-7da5-4863-90d9-c736126957ff-model-cache\") pod \"15c055e0-7da5-4863-90d9-c736126957ff\" (UID: \"15c055e0-7da5-4863-90d9-c736126957ff\") "
Sep 17 21:03:24.470931 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:03:24.470773    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gzj4l\" (UniqueName: \"kubernetes.io/projected/15c055e0-7da5-4863-90d9-c736126957ff-kube-api-access-gzj4l\") pod \"15c055e0-7da5-4863-90d9-c736126957ff\" (UID: \"15c055e0-7da5-4863-90d9-c736126957ff\") "
Sep 17 21:03:24.473185 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:03:24.473158    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15c055e0-7da5-4863-90d9-c736126957ff-kube-api-access-gzj4l" (OuterVolumeSpecName: "kube-api-access-gzj4l") pod "15c055e0-7da5-4863-90d9-c736126957ff" (UID: "15c055e0-7da5-4863-90d9-c736126957ff"). InnerVolumeSpecName "kube-api-access-gzj4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:03:24.489960 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:03:24.489926    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15c055e0-7da5-4863-90d9-c736126957ff-model-cache" (OuterVolumeSpecName: "model-cache") pod "15c055e0-7da5-4863-90d9-c736126957ff" (UID: "15c055e0-7da5-4863-90d9-c736126957ff"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:03:24.572110 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:03:24.572080    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15c055e0-7da5-4863-90d9-c736126957ff-model-cache\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:03:24.572110 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:03:24.572106    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzj4l\" (UniqueName: \"kubernetes.io/projected/15c055e0-7da5-4863-90d9-c736126957ff-kube-api-access-gzj4l\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:03:25.192475 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:03:25.192442    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-b9xfq"
Sep 17 21:03:25.192475 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:03:25.192450    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-b9xfq" event={"ID":"15c055e0-7da5-4863-90d9-c736126957ff","Type":"ContainerDied","Data":"987d83a288344e2a0b3fa603edd035e481cb033338463b2a9ec88757ec12fe09"}
Sep 17 21:03:25.192683 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:03:25.192495    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="987d83a288344e2a0b3fa603edd035e481cb033338463b2a9ec88757ec12fe09"
Sep 17 21:03:25.580409 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:03:25.580376    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:03:25.678352 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:03:25.678314    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:03:38.699809 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:03:38.699755    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:03:38.919672 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:03:38.919634    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:03:50.424717 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:03:50.424641    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:03:50.871013 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:03:50.870978    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:04:00.372917 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:00.372824    2575 generic.go:356] "Generic (PLEG): container finished" podID="bc057dcf-d949-4624-b5c7-631b60bbc740" containerID="4b7fe4a3031135f71716fc8ff25e9183762b05fbed00f36806f765ef4ed39c36" exitCode=0
Sep 17 21:04:00.372917 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:00.372865    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-92k25" event={"ID":"bc057dcf-d949-4624-b5c7-631b60bbc740","Type":"ContainerDied","Data":"4b7fe4a3031135f71716fc8ff25e9183762b05fbed00f36806f765ef4ed39c36"}
Sep 17 21:04:00.373347 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:00.373261    2575 scope.go:117] "RemoveContainer" containerID="4b7fe4a3031135f71716fc8ff25e9183762b05fbed00f36806f765ef4ed39c36"
Sep 17 21:04:00.374267 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:00.374239    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 21:04:00.374333 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:00.374306    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 21:04:00.374374 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:00.374334    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 21:04:01.379415 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:01.379380    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-92k25" event={"ID":"bc057dcf-d949-4624-b5c7-631b60bbc740","Type":"ContainerStarted","Data":"0bf3f239150235903eaab4cfa690257262d60fc3699a20c7f4dc8a50cc0f749d"}
Sep 17 21:04:01.546057 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:01.546017    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-92k25"
Sep 17 21:04:01.546300 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:01.546279    2575 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-92k25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body=
Sep 17 21:04:01.546353 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:01.546334    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-92k25" podUID="bc057dcf-d949-4624-b5c7-631b60bbc740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Sep 17 21:04:02.546474 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:02.546440    2575 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-92k25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body=
Sep 17 21:04:02.546923 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:02.546495    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-92k25" podUID="bc057dcf-d949-4624-b5c7-631b60bbc740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Sep 17 21:04:02.634313 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:02.634278    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-vb8t8"]
Sep 17 21:04:02.634677 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:02.634661    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15c055e0-7da5-4863-90d9-c736126957ff" containerName="s3-init"
Sep 17 21:04:02.634726 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:02.634680    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="15c055e0-7da5-4863-90d9-c736126957ff" containerName="s3-init"
Sep 17 21:04:02.634726 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:02.634698    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15c055e0-7da5-4863-90d9-c736126957ff" containerName="download-hf-model"
Sep 17 21:04:02.634726 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:02.634704    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="15c055e0-7da5-4863-90d9-c736126957ff" containerName="download-hf-model"
Sep 17 21:04:02.634819 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:02.634782    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="15c055e0-7da5-4863-90d9-c736126957ff" containerName="s3-init"
Sep 17 21:04:02.637831 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:02.637811    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-vb8t8"
Sep 17 21:04:02.640088 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:02.640063    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-ndgtw\""
Sep 17 21:04:02.641662 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:02.641642    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 17 21:04:02.642023 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:02.642007    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 17 21:04:02.642023 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:02.642014    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 17 21:04:02.651473 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:02.651449    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-vb8t8"]
Sep 17 21:04:02.659841 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:02.659810    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:04:02.717347 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:02.717304    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98h6z\" (UniqueName: \"kubernetes.io/projected/b0e87b6a-71c2-48a1-8c59-a4a99f7b106b-kube-api-access-98h6z\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-vb8t8\" (UID: \"b0e87b6a-71c2-48a1-8c59-a4a99f7b106b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-vb8t8"
Sep 17 21:04:02.717684 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:02.717659    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0e87b6a-71c2-48a1-8c59-a4a99f7b106b-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-vb8t8\" (UID: \"b0e87b6a-71c2-48a1-8c59-a4a99f7b106b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-vb8t8"
Sep 17 21:04:02.717904 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:02.717886    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0e87b6a-71c2-48a1-8c59-a4a99f7b106b-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-vb8t8\" (UID: \"b0e87b6a-71c2-48a1-8c59-a4a99f7b106b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-vb8t8"
Sep 17 21:04:02.718085 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:02.718066    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0e87b6a-71c2-48a1-8c59-a4a99f7b106b-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-vb8t8\" (UID: \"b0e87b6a-71c2-48a1-8c59-a4a99f7b106b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-vb8t8"
Sep 17 21:04:02.718302 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:02.718285    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0e87b6a-71c2-48a1-8c59-a4a99f7b106b-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-vb8t8\" (UID: \"b0e87b6a-71c2-48a1-8c59-a4a99f7b106b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-vb8t8"
Sep 17 21:04:02.718466 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:02.718451    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0e87b6a-71c2-48a1-8c59-a4a99f7b106b-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-vb8t8\" (UID: \"b0e87b6a-71c2-48a1-8c59-a4a99f7b106b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-vb8t8"
Sep 17 21:04:02.819635 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:02.819519    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0e87b6a-71c2-48a1-8c59-a4a99f7b106b-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-vb8t8\" (UID: \"b0e87b6a-71c2-48a1-8c59-a4a99f7b106b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-vb8t8"
Sep 17 21:04:02.819635 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:02.819608    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0e87b6a-71c2-48a1-8c59-a4a99f7b106b-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-vb8t8\" (UID: \"b0e87b6a-71c2-48a1-8c59-a4a99f7b106b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-vb8t8"
Sep 17 21:04:02.819896 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:02.819655    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98h6z\" (UniqueName: \"kubernetes.io/projected/b0e87b6a-71c2-48a1-8c59-a4a99f7b106b-kube-api-access-98h6z\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-vb8t8\" (UID: \"b0e87b6a-71c2-48a1-8c59-a4a99f7b106b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-vb8t8"
Sep 17 21:04:02.819896 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:02.819700    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0e87b6a-71c2-48a1-8c59-a4a99f7b106b-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-vb8t8\" (UID: \"b0e87b6a-71c2-48a1-8c59-a4a99f7b106b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-vb8t8"
Sep 17 21:04:02.819896 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:02.819744    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0e87b6a-71c2-48a1-8c59-a4a99f7b106b-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-vb8t8\" (UID: \"b0e87b6a-71c2-48a1-8c59-a4a99f7b106b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-vb8t8"
Sep 17 21:04:02.819896 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:02.819789    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0e87b6a-71c2-48a1-8c59-a4a99f7b106b-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-vb8t8\" (UID: \"b0e87b6a-71c2-48a1-8c59-a4a99f7b106b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-vb8t8"
Sep 17 21:04:02.820963 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:02.820915    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0e87b6a-71c2-48a1-8c59-a4a99f7b106b-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-vb8t8\" (UID: \"b0e87b6a-71c2-48a1-8c59-a4a99f7b106b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-vb8t8"
Sep 17 21:04:02.821419 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:02.821400    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0e87b6a-71c2-48a1-8c59-a4a99f7b106b-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-vb8t8\" (UID: \"b0e87b6a-71c2-48a1-8c59-a4a99f7b106b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-vb8t8"
Sep 17 21:04:02.821771 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:02.821751    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0e87b6a-71c2-48a1-8c59-a4a99f7b106b-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-vb8t8\" (UID: \"b0e87b6a-71c2-48a1-8c59-a4a99f7b106b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-vb8t8"
Sep 17 21:04:02.824107 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:02.824082    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0e87b6a-71c2-48a1-8c59-a4a99f7b106b-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-vb8t8\" (UID: \"b0e87b6a-71c2-48a1-8c59-a4a99f7b106b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-vb8t8"
Sep 17 21:04:02.825257 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:02.825230    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0e87b6a-71c2-48a1-8c59-a4a99f7b106b-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-vb8t8\" (UID: \"b0e87b6a-71c2-48a1-8c59-a4a99f7b106b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-vb8t8"
Sep 17 21:04:02.833729 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:02.833703    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98h6z\" (UniqueName: \"kubernetes.io/projected/b0e87b6a-71c2-48a1-8c59-a4a99f7b106b-kube-api-access-98h6z\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-vb8t8\" (UID: \"b0e87b6a-71c2-48a1-8c59-a4a99f7b106b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-vb8t8"
Sep 17 21:04:02.916875 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:02.915724    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:04:02.949418 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:02.949379    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-vb8t8"
Sep 17 21:04:03.239284 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:03.239261    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-vb8t8"]
Sep 17 21:04:03.241177 ip-10-0-139-36 kubenswrapper[2575]: W0917 21:04:03.241156    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0e87b6a_71c2_48a1_8c59_a4a99f7b106b.slice/crio-b1da8bb01140afd652480b8e105ec855ef0715c400fb2bbb4f84fa182862a594 WatchSource:0}: Error finding container b1da8bb01140afd652480b8e105ec855ef0715c400fb2bbb4f84fa182862a594: Status 404 returned error can't find the container with id b1da8bb01140afd652480b8e105ec855ef0715c400fb2bbb4f84fa182862a594
Sep 17 21:04:03.392317 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:03.392270    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-vb8t8" event={"ID":"b0e87b6a-71c2-48a1-8c59-a4a99f7b106b","Type":"ContainerStarted","Data":"b1da8bb01140afd652480b8e105ec855ef0715c400fb2bbb4f84fa182862a594"}
Sep 17 21:04:03.548577 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:03.548444    2575 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-92k25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body=
Sep 17 21:04:03.548577 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:03.548509    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-92k25" podUID="bc057dcf-d949-4624-b5c7-631b60bbc740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Sep 17 21:04:04.546889 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:04.546824    2575 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-92k25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body=
Sep 17 21:04:04.547078 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:04.546913    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-92k25" podUID="bc057dcf-d949-4624-b5c7-631b60bbc740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Sep 17 21:04:05.428671 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:05.428628    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-vb8t8" event={"ID":"b0e87b6a-71c2-48a1-8c59-a4a99f7b106b","Type":"ContainerStarted","Data":"debbceeb5834a8a38812e480ce58393ef8a2ad0e9f7d5166b524926714451f87"}
Sep 17 21:04:05.446908 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:05.446824    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-vb8t8" podStartSLOduration=1.392962313 podStartE2EDuration="3.446805458s" totalImagesPullingTime="2.053843145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:04:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:04:05.446275757 +0000 UTC m=+910.598720155" watchObservedRunningTime="2026-09-17 21:04:05.446805458 +0000 UTC m=+910.599249837"
Sep 17 21:04:05.546132 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:05.546056    2575 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-92k25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body=
Sep 17 21:04:05.546132 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:05.546113    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-92k25" podUID="bc057dcf-d949-4624-b5c7-631b60bbc740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Sep 17 21:04:05.949831 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:05.949797    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-vb8t8"
Sep 17 21:04:06.546116 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:06.546080    2575 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-92k25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body=
Sep 17 21:04:06.546506 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:06.546140    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-92k25" podUID="bc057dcf-d949-4624-b5c7-631b60bbc740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Sep 17 21:04:07.550651 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:07.550622    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-92k25"
Sep 17 21:04:07.551138 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:07.551057    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-92k25"
Sep 17 21:04:07.551785 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:07.551764    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-92k25"
Sep 17 21:04:12.949706 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:12.949665    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-vb8t8"
Sep 17 21:04:12.962557 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:12.962529    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-vb8t8"
Sep 17 21:04:13.484304 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:13.484276    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-vb8t8"
Sep 17 21:04:18.324474 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:18.324431    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:04:28.125287 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:28.125250    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:04:36.299547 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:36.299508    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:04:36.391278 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:36.391234    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:04:38.421024 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:38.420981    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-vb8t8"]
Sep 17 21:04:38.430420 ip-10-0-139-36 kubenswrapper[2575]: E0917 21:04:38.430386    2575 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 21:04:38.430420 ip-10-0-139-36 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:04:38.430420 ip-10-0-139-36 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 17 21:04:38.430420 ip-10-0-139-36 kubenswrapper[2575]:  > containerID="debbceeb5834a8a38812e480ce58393ef8a2ad0e9f7d5166b524926714451f87" cmd=["/bin/sleep","15"]
Sep 17 21:04:38.430789 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:38.430441    2575 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-vb8t8" message="" err=<
Sep 17 21:04:38.430789 ip-10-0-139-36 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:04:38.430789 ip-10-0-139-36 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 17 21:04:38.430789 ip-10-0-139-36 kubenswrapper[2575]:  >
Sep 17 21:04:38.430789 ip-10-0-139-36 kubenswrapper[2575]: E0917 21:04:38.430452    2575 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 21:04:38.430789 ip-10-0-139-36 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:04:38.430789 ip-10-0-139-36 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 17 21:04:38.430789 ip-10-0-139-36 kubenswrapper[2575]:  > pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-vb8t8" podUID="b0e87b6a-71c2-48a1-8c59-a4a99f7b106b" containerName="main" containerID="cri-o://debbceeb5834a8a38812e480ce58393ef8a2ad0e9f7d5166b524926714451f87"
Sep 17 21:04:38.430789 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:38.430499    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-vb8t8" podUID="b0e87b6a-71c2-48a1-8c59-a4a99f7b106b" containerName="main" containerID="cri-o://debbceeb5834a8a38812e480ce58393ef8a2ad0e9f7d5166b524926714451f87" gracePeriod=60
Sep 17 21:04:38.432282 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:38.432252    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:04:38.473461 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:38.473428    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-vb8t8" podUID="b0e87b6a-71c2-48a1-8c59-a4a99f7b106b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
Sep 17 21:04:38.543495 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:38.543461    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:04:38.778615 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:38.778594    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-vb8t8"
Sep 17 21:04:38.831547 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:38.831518    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0e87b6a-71c2-48a1-8c59-a4a99f7b106b-home\") pod \"b0e87b6a-71c2-48a1-8c59-a4a99f7b106b\" (UID: \"b0e87b6a-71c2-48a1-8c59-a4a99f7b106b\") "
Sep 17 21:04:38.831683 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:38.831570    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0e87b6a-71c2-48a1-8c59-a4a99f7b106b-dshm\") pod \"b0e87b6a-71c2-48a1-8c59-a4a99f7b106b\" (UID: \"b0e87b6a-71c2-48a1-8c59-a4a99f7b106b\") "
Sep 17 21:04:38.831683 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:38.831639    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0e87b6a-71c2-48a1-8c59-a4a99f7b106b-tmp-dir\") pod \"b0e87b6a-71c2-48a1-8c59-a4a99f7b106b\" (UID: \"b0e87b6a-71c2-48a1-8c59-a4a99f7b106b\") "
Sep 17 21:04:38.831683 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:38.831674    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-98h6z\" (UniqueName: \"kubernetes.io/projected/b0e87b6a-71c2-48a1-8c59-a4a99f7b106b-kube-api-access-98h6z\") pod \"b0e87b6a-71c2-48a1-8c59-a4a99f7b106b\" (UID: \"b0e87b6a-71c2-48a1-8c59-a4a99f7b106b\") "
Sep 17 21:04:38.831841 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:38.831699    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0e87b6a-71c2-48a1-8c59-a4a99f7b106b-tls-certs\") pod \"b0e87b6a-71c2-48a1-8c59-a4a99f7b106b\" (UID: \"b0e87b6a-71c2-48a1-8c59-a4a99f7b106b\") "
Sep 17 21:04:38.831841 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:38.831759    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0e87b6a-71c2-48a1-8c59-a4a99f7b106b-model-cache\") pod \"b0e87b6a-71c2-48a1-8c59-a4a99f7b106b\" (UID: \"b0e87b6a-71c2-48a1-8c59-a4a99f7b106b\") "
Sep 17 21:04:38.831841 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:38.831794    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0e87b6a-71c2-48a1-8c59-a4a99f7b106b-home" (OuterVolumeSpecName: "home") pod "b0e87b6a-71c2-48a1-8c59-a4a99f7b106b" (UID: "b0e87b6a-71c2-48a1-8c59-a4a99f7b106b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:04:38.832038 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:38.831943    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0e87b6a-71c2-48a1-8c59-a4a99f7b106b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b0e87b6a-71c2-48a1-8c59-a4a99f7b106b" (UID: "b0e87b6a-71c2-48a1-8c59-a4a99f7b106b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:04:38.832038 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:38.832028    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0e87b6a-71c2-48a1-8c59-a4a99f7b106b-model-cache" (OuterVolumeSpecName: "model-cache") pod "b0e87b6a-71c2-48a1-8c59-a4a99f7b106b" (UID: "b0e87b6a-71c2-48a1-8c59-a4a99f7b106b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:04:38.832150 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:38.832115    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0e87b6a-71c2-48a1-8c59-a4a99f7b106b-model-cache\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:04:38.832150 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:38.832130    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0e87b6a-71c2-48a1-8c59-a4a99f7b106b-home\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:04:38.832231 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:38.832165    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0e87b6a-71c2-48a1-8c59-a4a99f7b106b-tmp-dir\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:04:38.833918 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:38.833892    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0e87b6a-71c2-48a1-8c59-a4a99f7b106b-kube-api-access-98h6z" (OuterVolumeSpecName: "kube-api-access-98h6z") pod "b0e87b6a-71c2-48a1-8c59-a4a99f7b106b" (UID: "b0e87b6a-71c2-48a1-8c59-a4a99f7b106b"). InnerVolumeSpecName "kube-api-access-98h6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:04:38.833918 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:38.833906    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0e87b6a-71c2-48a1-8c59-a4a99f7b106b-dshm" (OuterVolumeSpecName: "dshm") pod "b0e87b6a-71c2-48a1-8c59-a4a99f7b106b" (UID: "b0e87b6a-71c2-48a1-8c59-a4a99f7b106b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:04:38.834177 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:38.834163    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0e87b6a-71c2-48a1-8c59-a4a99f7b106b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b0e87b6a-71c2-48a1-8c59-a4a99f7b106b" (UID: "b0e87b6a-71c2-48a1-8c59-a4a99f7b106b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:04:38.932880 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:38.932837    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98h6z\" (UniqueName: \"kubernetes.io/projected/b0e87b6a-71c2-48a1-8c59-a4a99f7b106b-kube-api-access-98h6z\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:04:38.932880 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:38.932878    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0e87b6a-71c2-48a1-8c59-a4a99f7b106b-tls-certs\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:04:38.933036 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:38.932889    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0e87b6a-71c2-48a1-8c59-a4a99f7b106b-dshm\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:04:39.600133 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:39.600092    2575 generic.go:356] "Generic (PLEG): container finished" podID="b0e87b6a-71c2-48a1-8c59-a4a99f7b106b" containerID="debbceeb5834a8a38812e480ce58393ef8a2ad0e9f7d5166b524926714451f87" exitCode=0
Sep 17 21:04:39.600606 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:39.600139    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-vb8t8"
Sep 17 21:04:39.600606 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:39.600181    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-vb8t8" event={"ID":"b0e87b6a-71c2-48a1-8c59-a4a99f7b106b","Type":"ContainerDied","Data":"debbceeb5834a8a38812e480ce58393ef8a2ad0e9f7d5166b524926714451f87"}
Sep 17 21:04:39.600606 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:39.600237    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-vb8t8" event={"ID":"b0e87b6a-71c2-48a1-8c59-a4a99f7b106b","Type":"ContainerDied","Data":"b1da8bb01140afd652480b8e105ec855ef0715c400fb2bbb4f84fa182862a594"}
Sep 17 21:04:39.600606 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:39.600260    2575 scope.go:117] "RemoveContainer" containerID="debbceeb5834a8a38812e480ce58393ef8a2ad0e9f7d5166b524926714451f87"
Sep 17 21:04:39.608930 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:39.608898    2575 scope.go:117] "RemoveContainer" containerID="debbceeb5834a8a38812e480ce58393ef8a2ad0e9f7d5166b524926714451f87"
Sep 17 21:04:39.609177 ip-10-0-139-36 kubenswrapper[2575]: E0917 21:04:39.609155    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"debbceeb5834a8a38812e480ce58393ef8a2ad0e9f7d5166b524926714451f87\": container with ID starting with debbceeb5834a8a38812e480ce58393ef8a2ad0e9f7d5166b524926714451f87 not found: ID does not exist" containerID="debbceeb5834a8a38812e480ce58393ef8a2ad0e9f7d5166b524926714451f87"
Sep 17 21:04:39.609224 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:39.609185    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"debbceeb5834a8a38812e480ce58393ef8a2ad0e9f7d5166b524926714451f87"} err="failed to get container status \"debbceeb5834a8a38812e480ce58393ef8a2ad0e9f7d5166b524926714451f87\": rpc error: code = NotFound desc = could not find container \"debbceeb5834a8a38812e480ce58393ef8a2ad0e9f7d5166b524926714451f87\": container with ID starting with debbceeb5834a8a38812e480ce58393ef8a2ad0e9f7d5166b524926714451f87 not found: ID does not exist"
Sep 17 21:04:39.617887 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:39.617798    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-vb8t8"]
Sep 17 21:04:39.619553 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:39.619535    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-vb8t8"]
Sep 17 21:04:41.356665 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:04:41.356632    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0e87b6a-71c2-48a1-8c59-a4a99f7b106b" path="/var/lib/kubelet/pods/b0e87b6a-71c2-48a1-8c59-a4a99f7b106b/volumes"
Sep 17 21:05:03.160412 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:05:03.160375    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:05:03.228049 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:05:03.228013    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:05:25.794103 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:05:25.794015    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:05:25.883482 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:05:25.883442    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:05:26.963723 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:05:26.963680    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:05:27.083133 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:05:27.083091    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:05:32.522409 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:05:32.522376    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:05:33.612893 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:05:33.612833    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:05:38.736877 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:05:38.736822    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:05:38.835045 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:05:38.835013    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:05:40.067957 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:05:40.067925    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:05:40.168390 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:05:40.168359    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:05:54.539732 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:05:54.539692    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:05:54.730193 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:05:54.730159    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:06:04.131928 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:06:04.131893    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:06:32.404025 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:06:32.403993    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6tq6l"]
Sep 17 21:06:32.404475 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:06:32.404377    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0e87b6a-71c2-48a1-8c59-a4a99f7b106b" containerName="main"
Sep 17 21:06:32.404475 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:06:32.404388    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0e87b6a-71c2-48a1-8c59-a4a99f7b106b" containerName="main"
Sep 17 21:06:32.404475 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:06:32.404461    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0e87b6a-71c2-48a1-8c59-a4a99f7b106b" containerName="main"
Sep 17 21:06:32.407064 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:06:32.407045    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6tq6l"
Sep 17 21:06:32.409271 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:06:32.409251    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Sep 17 21:06:32.409377 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:06:32.409322    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-blcmj\""
Sep 17 21:06:32.410122 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:06:32.410092    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Sep 17 21:06:32.410218 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:06:32.410158    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Sep 17 21:06:32.417438 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:06:32.417417    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6tq6l"]
Sep 17 21:06:32.418202 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:06:32.418181    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2af3274-f099-433f-96de-96d7efd46320-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-6tq6l\" (UID: \"a2af3274-f099-433f-96de-96d7efd46320\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6tq6l"
Sep 17 21:06:32.418282 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:06:32.418210    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2af3274-f099-433f-96de-96d7efd46320-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-6tq6l\" (UID: \"a2af3274-f099-433f-96de-96d7efd46320\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6tq6l"
Sep 17 21:06:32.418320 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:06:32.418281    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2af3274-f099-433f-96de-96d7efd46320-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-6tq6l\" (UID: \"a2af3274-f099-433f-96de-96d7efd46320\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6tq6l"
Sep 17 21:06:32.418320 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:06:32.418310    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2af3274-f099-433f-96de-96d7efd46320-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-6tq6l\" (UID: \"a2af3274-f099-433f-96de-96d7efd46320\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6tq6l"
Sep 17 21:06:32.418391 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:06:32.418333    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2af3274-f099-433f-96de-96d7efd46320-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-6tq6l\" (UID: \"a2af3274-f099-433f-96de-96d7efd46320\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6tq6l"
Sep 17 21:06:32.418425 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:06:32.418411    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm4rh\" (UniqueName: \"kubernetes.io/projected/a2af3274-f099-433f-96de-96d7efd46320-kube-api-access-hm4rh\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-6tq6l\" (UID: \"a2af3274-f099-433f-96de-96d7efd46320\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6tq6l"
Sep 17 21:06:32.437796 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:06:32.437746    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:06:32.519165 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:06:32.519128    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2af3274-f099-433f-96de-96d7efd46320-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-6tq6l\" (UID: \"a2af3274-f099-433f-96de-96d7efd46320\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6tq6l"
Sep 17 21:06:32.519165 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:06:32.519165    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2af3274-f099-433f-96de-96d7efd46320-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-6tq6l\" (UID: \"a2af3274-f099-433f-96de-96d7efd46320\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6tq6l"
Sep 17 21:06:32.519428 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:06:32.519211    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2af3274-f099-433f-96de-96d7efd46320-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-6tq6l\" (UID: \"a2af3274-f099-433f-96de-96d7efd46320\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6tq6l"
Sep 17 21:06:32.519428 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:06:32.519228    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2af3274-f099-433f-96de-96d7efd46320-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-6tq6l\" (UID: \"a2af3274-f099-433f-96de-96d7efd46320\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6tq6l"
Sep 17 21:06:32.519428 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:06:32.519258    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2af3274-f099-433f-96de-96d7efd46320-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-6tq6l\" (UID: \"a2af3274-f099-433f-96de-96d7efd46320\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6tq6l"
Sep 17 21:06:32.519428 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:06:32.519316    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm4rh\" (UniqueName: \"kubernetes.io/projected/a2af3274-f099-433f-96de-96d7efd46320-kube-api-access-hm4rh\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-6tq6l\" (UID: \"a2af3274-f099-433f-96de-96d7efd46320\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6tq6l"
Sep 17 21:06:32.519669 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:06:32.519617    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2af3274-f099-433f-96de-96d7efd46320-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-6tq6l\" (UID: \"a2af3274-f099-433f-96de-96d7efd46320\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6tq6l"
Sep 17 21:06:32.519727 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:06:32.519674    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2af3274-f099-433f-96de-96d7efd46320-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-6tq6l\" (UID: \"a2af3274-f099-433f-96de-96d7efd46320\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6tq6l"
Sep 17 21:06:32.519788 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:06:32.519743    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2af3274-f099-433f-96de-96d7efd46320-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-6tq6l\" (UID: \"a2af3274-f099-433f-96de-96d7efd46320\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6tq6l"
Sep 17 21:06:32.521477 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:06:32.521453    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2af3274-f099-433f-96de-96d7efd46320-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-6tq6l\" (UID: \"a2af3274-f099-433f-96de-96d7efd46320\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6tq6l"
Sep 17 21:06:32.521668 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:06:32.521654    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2af3274-f099-433f-96de-96d7efd46320-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-6tq6l\" (UID: \"a2af3274-f099-433f-96de-96d7efd46320\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6tq6l"
Sep 17 21:06:32.527664 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:06:32.527641    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm4rh\" (UniqueName: \"kubernetes.io/projected/a2af3274-f099-433f-96de-96d7efd46320-kube-api-access-hm4rh\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-6tq6l\" (UID: \"a2af3274-f099-433f-96de-96d7efd46320\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6tq6l"
Sep 17 21:06:32.718597 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:06:32.718558    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6tq6l"
Sep 17 21:06:33.004407 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:06:33.004376    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6tq6l"]
Sep 17 21:06:33.005991 ip-10-0-139-36 kubenswrapper[2575]: W0917 21:06:33.005965    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2af3274_f099_433f_96de_96d7efd46320.slice/crio-3da2c9569b26b4395336296255665a47dbb65b4990ef66b73bd9a86901859245 WatchSource:0}: Error finding container 3da2c9569b26b4395336296255665a47dbb65b4990ef66b73bd9a86901859245: Status 404 returned error can't find the container with id 3da2c9569b26b4395336296255665a47dbb65b4990ef66b73bd9a86901859245
Sep 17 21:06:33.167156 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:06:33.167124    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6tq6l" event={"ID":"a2af3274-f099-433f-96de-96d7efd46320","Type":"ContainerStarted","Data":"3bd9ad5b2fbd61ab89304bf6b3840dba576474fbb1be17d029607ae6e7df4217"}
Sep 17 21:06:33.167331 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:06:33.167179    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6tq6l" event={"ID":"a2af3274-f099-433f-96de-96d7efd46320","Type":"ContainerStarted","Data":"3da2c9569b26b4395336296255665a47dbb65b4990ef66b73bd9a86901859245"}
Sep 17 21:06:33.197862 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:06:33.197799    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6tq6l" podStartSLOduration=1.197782706 podStartE2EDuration="1.197782706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:06:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:06:33.196133179 +0000 UTC m=+1058.348577554" watchObservedRunningTime="2026-09-17 21:06:33.197782706 +0000 UTC m=+1058.350227123"
Sep 17 21:06:33.719208 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:06:33.719163    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6tq6l"
Sep 17 21:06:34.019421 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:06:34.019344    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:06:42.719763 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:06:42.719725    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6tq6l"
Sep 17 21:06:42.732532 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:06:42.732506    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6tq6l"
Sep 17 21:06:43.228109 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:06:43.228077    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6tq6l"
Sep 17 21:06:43.907928 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:06:43.907893    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:06:56.435607 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:06:56.435571    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:06:57.829615 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:06:57.829573    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:06:57.895204 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:06:57.895158    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:07:06.905953 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:06.905914    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6tq6l"]
Sep 17 21:07:06.909533 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:06.909499    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:07:06.918687 ip-10-0-139-36 kubenswrapper[2575]: E0917 21:07:06.918646    2575 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 21:07:06.918687 ip-10-0-139-36 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:07:06.918687 ip-10-0-139-36 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 17 21:07:06.918687 ip-10-0-139-36 kubenswrapper[2575]:  > containerID="3bd9ad5b2fbd61ab89304bf6b3840dba576474fbb1be17d029607ae6e7df4217" cmd=["/bin/sleep","15"]
Sep 17 21:07:06.919041 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:06.918711    2575 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6tq6l" message="" err=<
Sep 17 21:07:06.919041 ip-10-0-139-36 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:07:06.919041 ip-10-0-139-36 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 17 21:07:06.919041 ip-10-0-139-36 kubenswrapper[2575]:  >
Sep 17 21:07:06.919041 ip-10-0-139-36 kubenswrapper[2575]: E0917 21:07:06.918729    2575 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 21:07:06.919041 ip-10-0-139-36 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:07:06.919041 ip-10-0-139-36 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 17 21:07:06.919041 ip-10-0-139-36 kubenswrapper[2575]:  > pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6tq6l" podUID="a2af3274-f099-433f-96de-96d7efd46320" containerName="main" containerID="cri-o://3bd9ad5b2fbd61ab89304bf6b3840dba576474fbb1be17d029607ae6e7df4217"
Sep 17 21:07:06.919041 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:06.918783    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6tq6l" podUID="a2af3274-f099-433f-96de-96d7efd46320" containerName="main" containerID="cri-o://3bd9ad5b2fbd61ab89304bf6b3840dba576474fbb1be17d029607ae6e7df4217" gracePeriod=60
Sep 17 21:07:07.002873 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:07.002803    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:07:07.278859 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:07.278819    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6tq6l"
Sep 17 21:07:07.338273 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:07.338243    2575 generic.go:356] "Generic (PLEG): container finished" podID="a2af3274-f099-433f-96de-96d7efd46320" containerID="3bd9ad5b2fbd61ab89304bf6b3840dba576474fbb1be17d029607ae6e7df4217" exitCode=0
Sep 17 21:07:07.338467 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:07.338294    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6tq6l"
Sep 17 21:07:07.338467 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:07.338319    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6tq6l" event={"ID":"a2af3274-f099-433f-96de-96d7efd46320","Type":"ContainerDied","Data":"3bd9ad5b2fbd61ab89304bf6b3840dba576474fbb1be17d029607ae6e7df4217"}
Sep 17 21:07:07.338467 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:07.338361    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6tq6l" event={"ID":"a2af3274-f099-433f-96de-96d7efd46320","Type":"ContainerDied","Data":"3da2c9569b26b4395336296255665a47dbb65b4990ef66b73bd9a86901859245"}
Sep 17 21:07:07.338467 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:07.338383    2575 scope.go:117] "RemoveContainer" containerID="3bd9ad5b2fbd61ab89304bf6b3840dba576474fbb1be17d029607ae6e7df4217"
Sep 17 21:07:07.347389 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:07.347369    2575 scope.go:117] "RemoveContainer" containerID="3bd9ad5b2fbd61ab89304bf6b3840dba576474fbb1be17d029607ae6e7df4217"
Sep 17 21:07:07.347672 ip-10-0-139-36 kubenswrapper[2575]: E0917 21:07:07.347654    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bd9ad5b2fbd61ab89304bf6b3840dba576474fbb1be17d029607ae6e7df4217\": container with ID starting with 3bd9ad5b2fbd61ab89304bf6b3840dba576474fbb1be17d029607ae6e7df4217 not found: ID does not exist" containerID="3bd9ad5b2fbd61ab89304bf6b3840dba576474fbb1be17d029607ae6e7df4217"
Sep 17 21:07:07.347724 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:07.347680    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bd9ad5b2fbd61ab89304bf6b3840dba576474fbb1be17d029607ae6e7df4217"} err="failed to get container status \"3bd9ad5b2fbd61ab89304bf6b3840dba576474fbb1be17d029607ae6e7df4217\": rpc error: code = NotFound desc = could not find container \"3bd9ad5b2fbd61ab89304bf6b3840dba576474fbb1be17d029607ae6e7df4217\": container with ID starting with 3bd9ad5b2fbd61ab89304bf6b3840dba576474fbb1be17d029607ae6e7df4217 not found: ID does not exist"
Sep 17 21:07:07.432874 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:07.432827    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2af3274-f099-433f-96de-96d7efd46320-tmp-dir\") pod \"a2af3274-f099-433f-96de-96d7efd46320\" (UID: \"a2af3274-f099-433f-96de-96d7efd46320\") "
Sep 17 21:07:07.433065 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:07.432893    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hm4rh\" (UniqueName: \"kubernetes.io/projected/a2af3274-f099-433f-96de-96d7efd46320-kube-api-access-hm4rh\") pod \"a2af3274-f099-433f-96de-96d7efd46320\" (UID: \"a2af3274-f099-433f-96de-96d7efd46320\") "
Sep 17 21:07:07.433065 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:07.432941    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2af3274-f099-433f-96de-96d7efd46320-dshm\") pod \"a2af3274-f099-433f-96de-96d7efd46320\" (UID: \"a2af3274-f099-433f-96de-96d7efd46320\") "
Sep 17 21:07:07.433065 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:07.433005    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2af3274-f099-433f-96de-96d7efd46320-tls-certs\") pod \"a2af3274-f099-433f-96de-96d7efd46320\" (UID: \"a2af3274-f099-433f-96de-96d7efd46320\") "
Sep 17 21:07:07.433065 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:07.433023    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2af3274-f099-433f-96de-96d7efd46320-home\") pod \"a2af3274-f099-433f-96de-96d7efd46320\" (UID: \"a2af3274-f099-433f-96de-96d7efd46320\") "
Sep 17 21:07:07.433065 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:07.433052    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2af3274-f099-433f-96de-96d7efd46320-model-cache\") pod \"a2af3274-f099-433f-96de-96d7efd46320\" (UID: \"a2af3274-f099-433f-96de-96d7efd46320\") "
Sep 17 21:07:07.433343 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:07.433128    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2af3274-f099-433f-96de-96d7efd46320-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a2af3274-f099-433f-96de-96d7efd46320" (UID: "a2af3274-f099-433f-96de-96d7efd46320"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:07:07.433343 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:07.433330    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2af3274-f099-433f-96de-96d7efd46320-home" (OuterVolumeSpecName: "home") pod "a2af3274-f099-433f-96de-96d7efd46320" (UID: "a2af3274-f099-433f-96de-96d7efd46320"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:07:07.433343 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:07.433337    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2af3274-f099-433f-96de-96d7efd46320-model-cache" (OuterVolumeSpecName: "model-cache") pod "a2af3274-f099-433f-96de-96d7efd46320" (UID: "a2af3274-f099-433f-96de-96d7efd46320"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:07:07.433484 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:07.433420    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2af3274-f099-433f-96de-96d7efd46320-tmp-dir\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:07:07.435329 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:07.435305    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2af3274-f099-433f-96de-96d7efd46320-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a2af3274-f099-433f-96de-96d7efd46320" (UID: "a2af3274-f099-433f-96de-96d7efd46320"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:07:07.435329 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:07.435318    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2af3274-f099-433f-96de-96d7efd46320-kube-api-access-hm4rh" (OuterVolumeSpecName: "kube-api-access-hm4rh") pod "a2af3274-f099-433f-96de-96d7efd46320" (UID: "a2af3274-f099-433f-96de-96d7efd46320"). InnerVolumeSpecName "kube-api-access-hm4rh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:07:07.435329 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:07.435317    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2af3274-f099-433f-96de-96d7efd46320-dshm" (OuterVolumeSpecName: "dshm") pod "a2af3274-f099-433f-96de-96d7efd46320" (UID: "a2af3274-f099-433f-96de-96d7efd46320"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:07:07.534951 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:07.534912    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hm4rh\" (UniqueName: \"kubernetes.io/projected/a2af3274-f099-433f-96de-96d7efd46320-kube-api-access-hm4rh\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:07:07.534951 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:07.534944    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2af3274-f099-433f-96de-96d7efd46320-dshm\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:07:07.534951 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:07.534953    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2af3274-f099-433f-96de-96d7efd46320-tls-certs\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:07:07.534951 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:07.534964    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2af3274-f099-433f-96de-96d7efd46320-home\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:07:07.535249 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:07.534975    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2af3274-f099-433f-96de-96d7efd46320-model-cache\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:07:07.661114 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:07.661081    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6tq6l"]
Sep 17 21:07:07.668660 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:07.668637    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6tq6l"]
Sep 17 21:07:08.217250 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:08.217207    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6tq6l" podUID="a2af3274-f099-433f-96de-96d7efd46320" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 17 21:07:09.356017 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:09.355984    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2af3274-f099-433f-96de-96d7efd46320" path="/var/lib/kubelet/pods/a2af3274-f099-433f-96de-96d7efd46320/volumes"
Sep 17 21:07:47.093336 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:47.093298    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:07:47.202148 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:47.202107    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:07:47.310691 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:47.310652    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:07:47.386818 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:47.386789    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ghzqj"]
Sep 17 21:07:47.387252 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:47.387234    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2af3274-f099-433f-96de-96d7efd46320" containerName="main"
Sep 17 21:07:47.387348 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:47.387255    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2af3274-f099-433f-96de-96d7efd46320" containerName="main"
Sep 17 21:07:47.387405 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:47.387356    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2af3274-f099-433f-96de-96d7efd46320" containerName="main"
Sep 17 21:07:47.390403 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:47.390383    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ghzqj"
Sep 17 21:07:47.392967 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:47.392946    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"router-gateway-1-openshift-default-dockercfg-xzzws\""
Sep 17 21:07:47.393075 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:47.393004    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"kube-root-ca.crt\""
Sep 17 21:07:47.393788 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:47.393771    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-gw-ca-root-cert\""
Sep 17 21:07:47.393788 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:47.393785    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-service-ca.crt\""
Sep 17 21:07:47.409877 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:47.409843    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ghzqj"]
Sep 17 21:07:47.432718 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:47.432679    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:07:47.476856 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:47.476820    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5-istio-token\") pod \"router-gateway-1-openshift-default-5b4d9458bd-ghzqj\" (UID: \"2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ghzqj"
Sep 17 21:07:47.476998 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:47.476875    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5-istio-envoy\") pod \"router-gateway-1-openshift-default-5b4d9458bd-ghzqj\" (UID: \"2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ghzqj"
Sep 17 21:07:47.476998 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:47.476916    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5b4d9458bd-ghzqj\" (UID: \"2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ghzqj"
Sep 17 21:07:47.476998 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:47.476960    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5-credential-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-ghzqj\" (UID: \"2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ghzqj"
Sep 17 21:07:47.477190 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:47.477035    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5-workload-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-ghzqj\" (UID: \"2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ghzqj"
Sep 17 21:07:47.477190 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:47.477070    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5-workload-certs\") pod \"router-gateway-1-openshift-default-5b4d9458bd-ghzqj\" (UID: \"2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ghzqj"
Sep 17 21:07:47.477190 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:47.477098    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5-istio-podinfo\") pod \"router-gateway-1-openshift-default-5b4d9458bd-ghzqj\" (UID: \"2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ghzqj"
Sep 17 21:07:47.477190 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:47.477121    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5-istio-data\") pod \"router-gateway-1-openshift-default-5b4d9458bd-ghzqj\" (UID: \"2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ghzqj"
Sep 17 21:07:47.477190 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:47.477166    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zl45\" (UniqueName: \"kubernetes.io/projected/2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5-kube-api-access-9zl45\") pod \"router-gateway-1-openshift-default-5b4d9458bd-ghzqj\" (UID: \"2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ghzqj"
Sep 17 21:07:47.529966 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:47.529937    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:07:47.577923 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:47.577899    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5-credential-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-ghzqj\" (UID: \"2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ghzqj"
Sep 17 21:07:47.578037 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:47.577943    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5-workload-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-ghzqj\" (UID: \"2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ghzqj"
Sep 17 21:07:47.578037 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:47.577960    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5-workload-certs\") pod \"router-gateway-1-openshift-default-5b4d9458bd-ghzqj\" (UID: \"2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ghzqj"
Sep 17 21:07:47.578037 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:47.577977    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5-istio-podinfo\") pod \"router-gateway-1-openshift-default-5b4d9458bd-ghzqj\" (UID: \"2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ghzqj"
Sep 17 21:07:47.578037 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:47.577996    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5-istio-data\") pod \"router-gateway-1-openshift-default-5b4d9458bd-ghzqj\" (UID: \"2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ghzqj"
Sep 17 21:07:47.578037 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:47.578012    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zl45\" (UniqueName: \"kubernetes.io/projected/2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5-kube-api-access-9zl45\") pod \"router-gateway-1-openshift-default-5b4d9458bd-ghzqj\" (UID: \"2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ghzqj"
Sep 17 21:07:47.578277 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:47.578058    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5-istio-token\") pod \"router-gateway-1-openshift-default-5b4d9458bd-ghzqj\" (UID: \"2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ghzqj"
Sep 17 21:07:47.578277 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:47.578098    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5-istio-envoy\") pod \"router-gateway-1-openshift-default-5b4d9458bd-ghzqj\" (UID: \"2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ghzqj"
Sep 17 21:07:47.578277 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:47.578142    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5b4d9458bd-ghzqj\" (UID: \"2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ghzqj"
Sep 17 21:07:47.578427 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:47.578341    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5-credential-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-ghzqj\" (UID: \"2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ghzqj"
Sep 17 21:07:47.578427 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:47.578396    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5-workload-certs\") pod \"router-gateway-1-openshift-default-5b4d9458bd-ghzqj\" (UID: \"2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ghzqj"
Sep 17 21:07:47.578519 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:47.578421    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5-workload-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-ghzqj\" (UID: \"2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ghzqj"
Sep 17 21:07:47.578587 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:47.578567    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5-istio-data\") pod \"router-gateway-1-openshift-default-5b4d9458bd-ghzqj\" (UID: \"2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ghzqj"
Sep 17 21:07:47.578820 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:47.578802    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5b4d9458bd-ghzqj\" (UID: \"2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ghzqj"
Sep 17 21:07:47.580328 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:47.580304    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5-istio-envoy\") pod \"router-gateway-1-openshift-default-5b4d9458bd-ghzqj\" (UID: \"2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ghzqj"
Sep 17 21:07:47.580637 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:47.580620    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5-istio-podinfo\") pod \"router-gateway-1-openshift-default-5b4d9458bd-ghzqj\" (UID: \"2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ghzqj"
Sep 17 21:07:47.587114 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:47.587090    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5-istio-token\") pod \"router-gateway-1-openshift-default-5b4d9458bd-ghzqj\" (UID: \"2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ghzqj"
Sep 17 21:07:47.587265 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:47.587247    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zl45\" (UniqueName: \"kubernetes.io/projected/2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5-kube-api-access-9zl45\") pod \"router-gateway-1-openshift-default-5b4d9458bd-ghzqj\" (UID: \"2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ghzqj"
Sep 17 21:07:47.700202 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:47.700177    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ghzqj"
Sep 17 21:07:47.978975 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:47.978947    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ghzqj"]
Sep 17 21:07:47.981590 ip-10-0-139-36 kubenswrapper[2575]: W0917 21:07:47.981559    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c9a8f1e_9d5c_4c56_851c_95a14cdf88f5.slice/crio-6ca621a76dde87f73272d8701520f1c08e76c5aa436e4c0c732fcbdbd95ff65f WatchSource:0}: Error finding container 6ca621a76dde87f73272d8701520f1c08e76c5aa436e4c0c732fcbdbd95ff65f: Status 404 returned error can't find the container with id 6ca621a76dde87f73272d8701520f1c08e76c5aa436e4c0c732fcbdbd95ff65f
Sep 17 21:07:47.983702 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:47.983673    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 21:07:47.983764 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:47.983735    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 21:07:47.983804 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:47.983764    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 21:07:48.536492 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:48.536456    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ghzqj" event={"ID":"2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5","Type":"ContainerStarted","Data":"b29cbb50cc42d40ceda053b473633ac304e51f902fea1477f8b30ef3a9b93829"}
Sep 17 21:07:48.536897 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:48.536508    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ghzqj" event={"ID":"2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5","Type":"ContainerStarted","Data":"6ca621a76dde87f73272d8701520f1c08e76c5aa436e4c0c732fcbdbd95ff65f"}
Sep 17 21:07:48.559516 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:48.559467    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ghzqj" podStartSLOduration=1.559453062 podStartE2EDuration="1.559453062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:07:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:07:48.556941239 +0000 UTC m=+1133.709385609" watchObservedRunningTime="2026-09-17 21:07:48.559453062 +0000 UTC m=+1133.711897438"
Sep 17 21:07:48.701165 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:48.701136    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ghzqj"
Sep 17 21:07:48.733103 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:48.733073    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:07:48.831203 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:48.831132    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:07:49.706256 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:49.706224    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ghzqj"
Sep 17 21:07:50.077937 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:50.077831    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:07:50.546784 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:50.546751    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ghzqj"
Sep 17 21:07:50.547897 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:50.547878    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ghzqj"
Sep 17 21:07:50.986570 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:50.986544    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:07:53.420950 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:53.420908    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:07:56.607890 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:56.607828    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:07:56.707650 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:56.707622    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:07:59.482143 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:59.482104    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:07:59.583111 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:07:59.583076    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:08:02.731691 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:02.731659    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:08:08.002899 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:08.002843    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l2h4f"]
Sep 17 21:08:08.007664 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:08.007639    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l2h4f"
Sep 17 21:08:08.012083 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:08.012058    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs\""
Sep 17 21:08:08.012226 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:08.012066    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"default-dockercfg-zgn5j\""
Sep 17 21:08:08.014978 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:08.014954    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l2h4f"]
Sep 17 21:08:08.034706 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:08.034673    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:08:08.053458 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:08.053429    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pq5w\" (UniqueName: \"kubernetes.io/projected/26e34e22-9f44-410e-8d65-3f3c5425bccc-kube-api-access-7pq5w\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l2h4f\" (UID: \"26e34e22-9f44-410e-8d65-3f3c5425bccc\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l2h4f"
Sep 17 21:08:08.053586 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:08.053481    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26e34e22-9f44-410e-8d65-3f3c5425bccc-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l2h4f\" (UID: \"26e34e22-9f44-410e-8d65-3f3c5425bccc\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l2h4f"
Sep 17 21:08:08.053662 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:08.053583    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26e34e22-9f44-410e-8d65-3f3c5425bccc-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l2h4f\" (UID: \"26e34e22-9f44-410e-8d65-3f3c5425bccc\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l2h4f"
Sep 17 21:08:08.053662 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:08.053639    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26e34e22-9f44-410e-8d65-3f3c5425bccc-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l2h4f\" (UID: \"26e34e22-9f44-410e-8d65-3f3c5425bccc\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l2h4f"
Sep 17 21:08:08.053775 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:08.053736    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26e34e22-9f44-410e-8d65-3f3c5425bccc-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l2h4f\" (UID: \"26e34e22-9f44-410e-8d65-3f3c5425bccc\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l2h4f"
Sep 17 21:08:08.053775 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:08.053766    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26e34e22-9f44-410e-8d65-3f3c5425bccc-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l2h4f\" (UID: \"26e34e22-9f44-410e-8d65-3f3c5425bccc\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l2h4f"
Sep 17 21:08:08.154239 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:08.154196    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26e34e22-9f44-410e-8d65-3f3c5425bccc-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l2h4f\" (UID: \"26e34e22-9f44-410e-8d65-3f3c5425bccc\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l2h4f"
Sep 17 21:08:08.154239 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:08.154245    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26e34e22-9f44-410e-8d65-3f3c5425bccc-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l2h4f\" (UID: \"26e34e22-9f44-410e-8d65-3f3c5425bccc\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l2h4f"
Sep 17 21:08:08.154487 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:08.154292    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26e34e22-9f44-410e-8d65-3f3c5425bccc-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l2h4f\" (UID: \"26e34e22-9f44-410e-8d65-3f3c5425bccc\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l2h4f"
Sep 17 21:08:08.154487 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:08.154310    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26e34e22-9f44-410e-8d65-3f3c5425bccc-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l2h4f\" (UID: \"26e34e22-9f44-410e-8d65-3f3c5425bccc\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l2h4f"
Sep 17 21:08:08.154487 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:08.154329    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pq5w\" (UniqueName: \"kubernetes.io/projected/26e34e22-9f44-410e-8d65-3f3c5425bccc-kube-api-access-7pq5w\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l2h4f\" (UID: \"26e34e22-9f44-410e-8d65-3f3c5425bccc\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l2h4f"
Sep 17 21:08:08.154487 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:08.154362    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26e34e22-9f44-410e-8d65-3f3c5425bccc-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l2h4f\" (UID: \"26e34e22-9f44-410e-8d65-3f3c5425bccc\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l2h4f"
Sep 17 21:08:08.154755 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:08.154729    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26e34e22-9f44-410e-8d65-3f3c5425bccc-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l2h4f\" (UID: \"26e34e22-9f44-410e-8d65-3f3c5425bccc\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l2h4f"
Sep 17 21:08:08.154834 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:08.154770    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26e34e22-9f44-410e-8d65-3f3c5425bccc-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l2h4f\" (UID: \"26e34e22-9f44-410e-8d65-3f3c5425bccc\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l2h4f"
Sep 17 21:08:08.154834 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:08.154791    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26e34e22-9f44-410e-8d65-3f3c5425bccc-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l2h4f\" (UID: \"26e34e22-9f44-410e-8d65-3f3c5425bccc\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l2h4f"
Sep 17 21:08:08.156640 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:08.156610    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26e34e22-9f44-410e-8d65-3f3c5425bccc-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l2h4f\" (UID: \"26e34e22-9f44-410e-8d65-3f3c5425bccc\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l2h4f"
Sep 17 21:08:08.156813 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:08.156793    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26e34e22-9f44-410e-8d65-3f3c5425bccc-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l2h4f\" (UID: \"26e34e22-9f44-410e-8d65-3f3c5425bccc\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l2h4f"
Sep 17 21:08:08.162035 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:08.162012    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pq5w\" (UniqueName: \"kubernetes.io/projected/26e34e22-9f44-410e-8d65-3f3c5425bccc-kube-api-access-7pq5w\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l2h4f\" (UID: \"26e34e22-9f44-410e-8d65-3f3c5425bccc\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l2h4f"
Sep 17 21:08:08.319440 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:08.319356    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l2h4f"
Sep 17 21:08:08.604045 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:08.603955    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l2h4f"]
Sep 17 21:08:08.616832 ip-10-0-139-36 kubenswrapper[2575]: W0917 21:08:08.616803    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26e34e22_9f44_410e_8d65_3f3c5425bccc.slice/crio-066f470b392ff6674967f153075c113b0558df47c31e93356e79f22c3f976bc7 WatchSource:0}: Error finding container 066f470b392ff6674967f153075c113b0558df47c31e93356e79f22c3f976bc7: Status 404 returned error can't find the container with id 066f470b392ff6674967f153075c113b0558df47c31e93356e79f22c3f976bc7
Sep 17 21:08:08.640193 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:08.640160    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l2h4f" event={"ID":"26e34e22-9f44-410e-8d65-3f3c5425bccc","Type":"ContainerStarted","Data":"066f470b392ff6674967f153075c113b0558df47c31e93356e79f22c3f976bc7"}
Sep 17 21:08:09.523064 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:09.523021    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:08:09.625573 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:09.625524    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:08:09.647086 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:09.647049    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l2h4f" event={"ID":"26e34e22-9f44-410e-8d65-3f3c5425bccc","Type":"ContainerStarted","Data":"2ab5a96ac80208dbc7aa217e90a7dc6590d6c1c720538d4a335ec22465eb9013"}
Sep 17 21:08:09.671023 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:09.670969    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l2h4f" podStartSLOduration=2.6709556230000002 podStartE2EDuration="2.670955623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:08:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:08:09.668468572 +0000 UTC m=+1154.820912947" watchObservedRunningTime="2026-09-17 21:08:09.670955623 +0000 UTC m=+1154.823399999"
Sep 17 21:08:10.319807 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:10.319772    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l2h4f"
Sep 17 21:08:11.125451 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:11.125409    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:08:11.225675 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:11.225629    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:08:11.327414 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:11.327378    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:08:11.427824 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:11.427792    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:08:11.492811 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:11.492763    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:08:14.509615 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:14.509576    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:08:16.107810 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:16.107776    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l2h4f"]
Sep 17 21:08:16.115659 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:16.115629    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ghzqj"]
Sep 17 21:08:16.115869 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:16.115824    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ghzqj" podUID="2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5" containerName="istio-proxy" containerID="cri-o://b29cbb50cc42d40ceda053b473633ac304e51f902fea1477f8b30ef3a9b93829" gracePeriod=30
Sep 17 21:08:16.117374 ip-10-0-139-36 kubenswrapper[2575]: E0917 21:08:16.117341    2575 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 21:08:16.117374 ip-10-0-139-36 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:08:16.117374 ip-10-0-139-36 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 17 21:08:16.117374 ip-10-0-139-36 kubenswrapper[2575]:  > containerID="2ab5a96ac80208dbc7aa217e90a7dc6590d6c1c720538d4a335ec22465eb9013" cmd=["/bin/sleep","15"]
Sep 17 21:08:16.117703 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:16.117392    2575 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l2h4f" message="" err=<
Sep 17 21:08:16.117703 ip-10-0-139-36 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:08:16.117703 ip-10-0-139-36 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 17 21:08:16.117703 ip-10-0-139-36 kubenswrapper[2575]:  >
Sep 17 21:08:16.117703 ip-10-0-139-36 kubenswrapper[2575]: E0917 21:08:16.117404    2575 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 21:08:16.117703 ip-10-0-139-36 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:08:16.117703 ip-10-0-139-36 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 17 21:08:16.117703 ip-10-0-139-36 kubenswrapper[2575]:  > pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l2h4f" podUID="26e34e22-9f44-410e-8d65-3f3c5425bccc" containerName="main" containerID="cri-o://2ab5a96ac80208dbc7aa217e90a7dc6590d6c1c720538d4a335ec22465eb9013"
Sep 17 21:08:16.117703 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:16.117436    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l2h4f" podUID="26e34e22-9f44-410e-8d65-3f3c5425bccc" containerName="main" containerID="cri-o://2ab5a96ac80208dbc7aa217e90a7dc6590d6c1c720538d4a335ec22465eb9013" gracePeriod=60
Sep 17 21:08:16.171081 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:16.171041    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:08:16.369325 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:16.369259    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l2h4f"
Sep 17 21:08:16.438230 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:16.438199    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7pq5w\" (UniqueName: \"kubernetes.io/projected/26e34e22-9f44-410e-8d65-3f3c5425bccc-kube-api-access-7pq5w\") pod \"26e34e22-9f44-410e-8d65-3f3c5425bccc\" (UID: \"26e34e22-9f44-410e-8d65-3f3c5425bccc\") "
Sep 17 21:08:16.438423 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:16.438293    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26e34e22-9f44-410e-8d65-3f3c5425bccc-model-cache\") pod \"26e34e22-9f44-410e-8d65-3f3c5425bccc\" (UID: \"26e34e22-9f44-410e-8d65-3f3c5425bccc\") "
Sep 17 21:08:16.438423 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:16.438333    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26e34e22-9f44-410e-8d65-3f3c5425bccc-tmp-dir\") pod \"26e34e22-9f44-410e-8d65-3f3c5425bccc\" (UID: \"26e34e22-9f44-410e-8d65-3f3c5425bccc\") "
Sep 17 21:08:16.438423 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:16.438349    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26e34e22-9f44-410e-8d65-3f3c5425bccc-dshm\") pod \"26e34e22-9f44-410e-8d65-3f3c5425bccc\" (UID: \"26e34e22-9f44-410e-8d65-3f3c5425bccc\") "
Sep 17 21:08:16.438423 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:16.438418    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26e34e22-9f44-410e-8d65-3f3c5425bccc-tls-certs\") pod \"26e34e22-9f44-410e-8d65-3f3c5425bccc\" (UID: \"26e34e22-9f44-410e-8d65-3f3c5425bccc\") "
Sep 17 21:08:16.438644 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:16.438455    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26e34e22-9f44-410e-8d65-3f3c5425bccc-home\") pod \"26e34e22-9f44-410e-8d65-3f3c5425bccc\" (UID: \"26e34e22-9f44-410e-8d65-3f3c5425bccc\") "
Sep 17 21:08:16.438644 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:16.438598    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26e34e22-9f44-410e-8d65-3f3c5425bccc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "26e34e22-9f44-410e-8d65-3f3c5425bccc" (UID: "26e34e22-9f44-410e-8d65-3f3c5425bccc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:08:16.438644 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:16.438604    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26e34e22-9f44-410e-8d65-3f3c5425bccc-model-cache" (OuterVolumeSpecName: "model-cache") pod "26e34e22-9f44-410e-8d65-3f3c5425bccc" (UID: "26e34e22-9f44-410e-8d65-3f3c5425bccc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:08:16.438902 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:16.438877    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26e34e22-9f44-410e-8d65-3f3c5425bccc-model-cache\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:08:16.439003 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:16.438905    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26e34e22-9f44-410e-8d65-3f3c5425bccc-tmp-dir\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:08:16.439003 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:16.438927    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26e34e22-9f44-410e-8d65-3f3c5425bccc-home" (OuterVolumeSpecName: "home") pod "26e34e22-9f44-410e-8d65-3f3c5425bccc" (UID: "26e34e22-9f44-410e-8d65-3f3c5425bccc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:08:16.440618 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:16.440592    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26e34e22-9f44-410e-8d65-3f3c5425bccc-dshm" (OuterVolumeSpecName: "dshm") pod "26e34e22-9f44-410e-8d65-3f3c5425bccc" (UID: "26e34e22-9f44-410e-8d65-3f3c5425bccc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:08:16.440714 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:16.440644    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26e34e22-9f44-410e-8d65-3f3c5425bccc-kube-api-access-7pq5w" (OuterVolumeSpecName: "kube-api-access-7pq5w") pod "26e34e22-9f44-410e-8d65-3f3c5425bccc" (UID: "26e34e22-9f44-410e-8d65-3f3c5425bccc"). InnerVolumeSpecName "kube-api-access-7pq5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:08:16.440965 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:16.440944    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26e34e22-9f44-410e-8d65-3f3c5425bccc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "26e34e22-9f44-410e-8d65-3f3c5425bccc" (UID: "26e34e22-9f44-410e-8d65-3f3c5425bccc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:08:16.539810 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:16.539770    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26e34e22-9f44-410e-8d65-3f3c5425bccc-dshm\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:08:16.539810 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:16.539801    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26e34e22-9f44-410e-8d65-3f3c5425bccc-tls-certs\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:08:16.539810 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:16.539812    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26e34e22-9f44-410e-8d65-3f3c5425bccc-home\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:08:16.540073 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:16.539821    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pq5w\" (UniqueName: \"kubernetes.io/projected/26e34e22-9f44-410e-8d65-3f3c5425bccc-kube-api-access-7pq5w\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:08:16.684218 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:16.684132    2575 generic.go:356] "Generic (PLEG): container finished" podID="26e34e22-9f44-410e-8d65-3f3c5425bccc" containerID="2ab5a96ac80208dbc7aa217e90a7dc6590d6c1c720538d4a335ec22465eb9013" exitCode=0
Sep 17 21:08:16.684218 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:16.684184    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l2h4f"
Sep 17 21:08:16.684442 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:16.684227    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l2h4f" event={"ID":"26e34e22-9f44-410e-8d65-3f3c5425bccc","Type":"ContainerDied","Data":"2ab5a96ac80208dbc7aa217e90a7dc6590d6c1c720538d4a335ec22465eb9013"}
Sep 17 21:08:16.684442 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:16.684301    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l2h4f" event={"ID":"26e34e22-9f44-410e-8d65-3f3c5425bccc","Type":"ContainerDied","Data":"066f470b392ff6674967f153075c113b0558df47c31e93356e79f22c3f976bc7"}
Sep 17 21:08:16.684442 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:16.684334    2575 scope.go:117] "RemoveContainer" containerID="2ab5a96ac80208dbc7aa217e90a7dc6590d6c1c720538d4a335ec22465eb9013"
Sep 17 21:08:16.695281 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:16.695264    2575 scope.go:117] "RemoveContainer" containerID="2ab5a96ac80208dbc7aa217e90a7dc6590d6c1c720538d4a335ec22465eb9013"
Sep 17 21:08:16.695584 ip-10-0-139-36 kubenswrapper[2575]: E0917 21:08:16.695562    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ab5a96ac80208dbc7aa217e90a7dc6590d6c1c720538d4a335ec22465eb9013\": container with ID starting with 2ab5a96ac80208dbc7aa217e90a7dc6590d6c1c720538d4a335ec22465eb9013 not found: ID does not exist" containerID="2ab5a96ac80208dbc7aa217e90a7dc6590d6c1c720538d4a335ec22465eb9013"
Sep 17 21:08:16.695645 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:16.695597    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ab5a96ac80208dbc7aa217e90a7dc6590d6c1c720538d4a335ec22465eb9013"} err="failed to get container status \"2ab5a96ac80208dbc7aa217e90a7dc6590d6c1c720538d4a335ec22465eb9013\": rpc error: code = NotFound desc = could not find container \"2ab5a96ac80208dbc7aa217e90a7dc6590d6c1c720538d4a335ec22465eb9013\": container with ID starting with 2ab5a96ac80208dbc7aa217e90a7dc6590d6c1c720538d4a335ec22465eb9013 not found: ID does not exist"
Sep 17 21:08:16.706086 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:16.706062    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l2h4f"]
Sep 17 21:08:16.710857 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:16.710821    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l2h4f"]
Sep 17 21:08:17.268201 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:17.268175    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ghzqj"
Sep 17 21:08:17.345374 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:17.345291    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5-istio-podinfo\") pod \"2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5\" (UID: \"2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5\") "
Sep 17 21:08:17.345374 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:17.345328    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9zl45\" (UniqueName: \"kubernetes.io/projected/2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5-kube-api-access-9zl45\") pod \"2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5\" (UID: \"2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5\") "
Sep 17 21:08:17.345604 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:17.345403    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5-workload-certs\") pod \"2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5\" (UID: \"2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5\") "
Sep 17 21:08:17.345604 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:17.345429    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5-istio-token\") pod \"2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5\" (UID: \"2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5\") "
Sep 17 21:08:17.345604 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:17.345451    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5-istio-data\") pod \"2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5\" (UID: \"2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5\") "
Sep 17 21:08:17.345604 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:17.345492    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5-istiod-ca-cert\") pod \"2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5\" (UID: \"2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5\") "
Sep 17 21:08:17.345604 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:17.345519    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5-istio-envoy\") pod \"2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5\" (UID: \"2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5\") "
Sep 17 21:08:17.345604 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:17.345541    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5-workload-socket\") pod \"2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5\" (UID: \"2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5\") "
Sep 17 21:08:17.346402 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:17.345638    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5-credential-socket\") pod \"2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5\" (UID: \"2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5\") "
Sep 17 21:08:17.346402 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:17.346027    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5" (UID: "2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:08:17.346402 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:17.346036    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5" (UID: "2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:08:17.346402 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:17.346154    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5" (UID: "2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:08:17.346402 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:17.346277    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5-istio-data" (OuterVolumeSpecName: "istio-data") pod "2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5" (UID: "2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:08:17.346402 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:17.346306    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5" (UID: "2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:08:17.348190 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:17.348153    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5-kube-api-access-9zl45" (OuterVolumeSpecName: "kube-api-access-9zl45") pod "2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5" (UID: "2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5"). InnerVolumeSpecName "kube-api-access-9zl45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:08:17.348301 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:17.348231    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5-istio-token" (OuterVolumeSpecName: "istio-token") pod "2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5" (UID: "2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:08:17.348401 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:17.348383    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5" (UID: "2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:08:17.348458 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:17.348393    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5" (UID: "2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 21:08:17.359134 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:17.359103    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26e34e22-9f44-410e-8d65-3f3c5425bccc" path="/var/lib/kubelet/pods/26e34e22-9f44-410e-8d65-3f3c5425bccc/volumes"
Sep 17 21:08:17.447340 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:17.447307    2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5-workload-certs\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:08:17.447340 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:17.447334    2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5-istio-token\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:08:17.447340 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:17.447342    2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5-istio-data\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:08:17.447340 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:17.447350    2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5-istiod-ca-cert\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:08:17.447611 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:17.447359    2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5-istio-envoy\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:08:17.447611 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:17.447367    2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5-workload-socket\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:08:17.447611 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:17.447374    2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5-credential-socket\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:08:17.447611 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:17.447386    2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5-istio-podinfo\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:08:17.447611 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:17.447395    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zl45\" (UniqueName: \"kubernetes.io/projected/2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5-kube-api-access-9zl45\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:08:17.690970 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:17.690884    2575 generic.go:356] "Generic (PLEG): container finished" podID="2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5" containerID="b29cbb50cc42d40ceda053b473633ac304e51f902fea1477f8b30ef3a9b93829" exitCode=0
Sep 17 21:08:17.690970 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:17.690949    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ghzqj" event={"ID":"2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5","Type":"ContainerDied","Data":"b29cbb50cc42d40ceda053b473633ac304e51f902fea1477f8b30ef3a9b93829"}
Sep 17 21:08:17.691173 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:17.690972    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ghzqj"
Sep 17 21:08:17.691173 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:17.690983    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ghzqj" event={"ID":"2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5","Type":"ContainerDied","Data":"6ca621a76dde87f73272d8701520f1c08e76c5aa436e4c0c732fcbdbd95ff65f"}
Sep 17 21:08:17.691173 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:17.691005    2575 scope.go:117] "RemoveContainer" containerID="b29cbb50cc42d40ceda053b473633ac304e51f902fea1477f8b30ef3a9b93829"
Sep 17 21:08:17.699860 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:17.699828    2575 scope.go:117] "RemoveContainer" containerID="b29cbb50cc42d40ceda053b473633ac304e51f902fea1477f8b30ef3a9b93829"
Sep 17 21:08:17.700094 ip-10-0-139-36 kubenswrapper[2575]: E0917 21:08:17.700075    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b29cbb50cc42d40ceda053b473633ac304e51f902fea1477f8b30ef3a9b93829\": container with ID starting with b29cbb50cc42d40ceda053b473633ac304e51f902fea1477f8b30ef3a9b93829 not found: ID does not exist" containerID="b29cbb50cc42d40ceda053b473633ac304e51f902fea1477f8b30ef3a9b93829"
Sep 17 21:08:17.700155 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:17.700101    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b29cbb50cc42d40ceda053b473633ac304e51f902fea1477f8b30ef3a9b93829"} err="failed to get container status \"b29cbb50cc42d40ceda053b473633ac304e51f902fea1477f8b30ef3a9b93829\": rpc error: code = NotFound desc = could not find container \"b29cbb50cc42d40ceda053b473633ac304e51f902fea1477f8b30ef3a9b93829\": container with ID starting with b29cbb50cc42d40ceda053b473633ac304e51f902fea1477f8b30ef3a9b93829 not found: ID does not exist"
Sep 17 21:08:17.709180 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:17.709158    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ghzqj"]
Sep 17 21:08:17.715884 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:17.715839    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ghzqj"]
Sep 17 21:08:19.357398 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:19.357363    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5" path="/var/lib/kubelet/pods/2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5/volumes"
Sep 17 21:08:21.381804 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:21.381767    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:08:21.437805 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:21.437771    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:08:23.534324 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:23.534296    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:08:27.834503 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:27.834468    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:08:29.320143 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:29.320089    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:08:55.751097 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:55.751051    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-k99td"]
Sep 17 21:08:55.751538 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:55.751528    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26e34e22-9f44-410e-8d65-3f3c5425bccc" containerName="main"
Sep 17 21:08:55.751618 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:55.751543    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="26e34e22-9f44-410e-8d65-3f3c5425bccc" containerName="main"
Sep 17 21:08:55.751618 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:55.751555    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5" containerName="istio-proxy"
Sep 17 21:08:55.751618 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:55.751561    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5" containerName="istio-proxy"
Sep 17 21:08:55.751781 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:55.751654    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c9a8f1e-9d5c-4c56-851c-95a14cdf88f5" containerName="istio-proxy"
Sep 17 21:08:55.751781 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:55.751667    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="26e34e22-9f44-410e-8d65-3f3c5425bccc" containerName="main"
Sep 17 21:08:55.756272 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:55.756251    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-k99td"
Sep 17 21:08:55.758830 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:55.758809    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Sep 17 21:08:55.758972 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:55.758835    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-j6wnd\""
Sep 17 21:08:55.759650 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:55.759630    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Sep 17 21:08:55.760308 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:55.760286    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-dockercfg-rtdgs\""
Sep 17 21:08:55.760640 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:55.760305    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Sep 17 21:08:55.769298 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:55.769095    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-k99td"]
Sep 17 21:08:55.771525 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:55.771501    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/598ca3a5-3638-4ff3-99a3-118ecd4590d5-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-k99td\" (UID: \"598ca3a5-3638-4ff3-99a3-118ecd4590d5\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-k99td"
Sep 17 21:08:55.771629 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:55.771563    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/598ca3a5-3638-4ff3-99a3-118ecd4590d5-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-k99td\" (UID: \"598ca3a5-3638-4ff3-99a3-118ecd4590d5\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-k99td"
Sep 17 21:08:55.771682 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:55.771655    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/598ca3a5-3638-4ff3-99a3-118ecd4590d5-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-k99td\" (UID: \"598ca3a5-3638-4ff3-99a3-118ecd4590d5\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-k99td"
Sep 17 21:08:55.771733 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:55.771685    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/598ca3a5-3638-4ff3-99a3-118ecd4590d5-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-k99td\" (UID: \"598ca3a5-3638-4ff3-99a3-118ecd4590d5\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-k99td"
Sep 17 21:08:55.771733 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:55.771710    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/598ca3a5-3638-4ff3-99a3-118ecd4590d5-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-k99td\" (UID: \"598ca3a5-3638-4ff3-99a3-118ecd4590d5\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-k99td"
Sep 17 21:08:55.771869 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:55.771803    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgt9m\" (UniqueName: \"kubernetes.io/projected/598ca3a5-3638-4ff3-99a3-118ecd4590d5-kube-api-access-pgt9m\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-k99td\" (UID: \"598ca3a5-3638-4ff3-99a3-118ecd4590d5\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-k99td"
Sep 17 21:08:55.788515 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:55.788489    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:08:55.872387 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:55.872363    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/598ca3a5-3638-4ff3-99a3-118ecd4590d5-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-k99td\" (UID: \"598ca3a5-3638-4ff3-99a3-118ecd4590d5\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-k99td"
Sep 17 21:08:55.872485 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:55.872398    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/598ca3a5-3638-4ff3-99a3-118ecd4590d5-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-k99td\" (UID: \"598ca3a5-3638-4ff3-99a3-118ecd4590d5\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-k99td"
Sep 17 21:08:55.872485 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:55.872428    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/598ca3a5-3638-4ff3-99a3-118ecd4590d5-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-k99td\" (UID: \"598ca3a5-3638-4ff3-99a3-118ecd4590d5\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-k99td"
Sep 17 21:08:55.872485 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:55.872472    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgt9m\" (UniqueName: \"kubernetes.io/projected/598ca3a5-3638-4ff3-99a3-118ecd4590d5-kube-api-access-pgt9m\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-k99td\" (UID: \"598ca3a5-3638-4ff3-99a3-118ecd4590d5\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-k99td"
Sep 17 21:08:55.872591 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:55.872538    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/598ca3a5-3638-4ff3-99a3-118ecd4590d5-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-k99td\" (UID: \"598ca3a5-3638-4ff3-99a3-118ecd4590d5\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-k99td"
Sep 17 21:08:55.872633 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:55.872595    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/598ca3a5-3638-4ff3-99a3-118ecd4590d5-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-k99td\" (UID: \"598ca3a5-3638-4ff3-99a3-118ecd4590d5\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-k99td"
Sep 17 21:08:55.872809 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:55.872788    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/598ca3a5-3638-4ff3-99a3-118ecd4590d5-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-k99td\" (UID: \"598ca3a5-3638-4ff3-99a3-118ecd4590d5\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-k99td"
Sep 17 21:08:55.872904 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:55.872829    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/598ca3a5-3638-4ff3-99a3-118ecd4590d5-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-k99td\" (UID: \"598ca3a5-3638-4ff3-99a3-118ecd4590d5\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-k99td"
Sep 17 21:08:55.872950 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:55.872904    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/598ca3a5-3638-4ff3-99a3-118ecd4590d5-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-k99td\" (UID: \"598ca3a5-3638-4ff3-99a3-118ecd4590d5\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-k99td"
Sep 17 21:08:55.874661 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:55.874640    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/598ca3a5-3638-4ff3-99a3-118ecd4590d5-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-k99td\" (UID: \"598ca3a5-3638-4ff3-99a3-118ecd4590d5\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-k99td"
Sep 17 21:08:55.875036 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:55.875015    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/598ca3a5-3638-4ff3-99a3-118ecd4590d5-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-k99td\" (UID: \"598ca3a5-3638-4ff3-99a3-118ecd4590d5\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-k99td"
Sep 17 21:08:55.881655 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:55.881635    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgt9m\" (UniqueName: \"kubernetes.io/projected/598ca3a5-3638-4ff3-99a3-118ecd4590d5-kube-api-access-pgt9m\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-k99td\" (UID: \"598ca3a5-3638-4ff3-99a3-118ecd4590d5\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-k99td"
Sep 17 21:08:55.995246 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:55.995179    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:08:56.071349 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:56.071264    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-k99td"
Sep 17 21:08:56.350529 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:56.350323    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-k99td"]
Sep 17 21:08:56.359025 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:56.357090    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 21:08:56.903390 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:56.903348    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-k99td" event={"ID":"598ca3a5-3638-4ff3-99a3-118ecd4590d5","Type":"ContainerStarted","Data":"dfdfa6a1be934d8eb81eb2ad821fab21d7376691e7c7072214373b69684a84a0"}
Sep 17 21:08:59.941645 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:59.941602    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-k99td" event={"ID":"598ca3a5-3638-4ff3-99a3-118ecd4590d5","Type":"ContainerStarted","Data":"39505a6bc54656497b841fdd1cd15e6f59ece8239501bf3648e9bed0c28144bf"}
Sep 17 21:08:59.942142 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:08:59.941668    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-k99td"
Sep 17 21:09:00.949155 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:09:00.949116    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-k99td" event={"ID":"598ca3a5-3638-4ff3-99a3-118ecd4590d5","Type":"ContainerStarted","Data":"888a4e2a54eea1d0f3c1d207dee99ea55e0c9c68c3371e47e1be31a2c75bc145"}
Sep 17 21:09:00.970442 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:09:00.970387    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-k99td" podStartSLOduration=2.944899119 podStartE2EDuration="5.970373023s" totalImagesPullingTime="3.025473904s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:08:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:09:00.968905596 +0000 UTC m=+1206.121349973" watchObservedRunningTime="2026-09-17 21:09:00.970373023 +0000 UTC m=+1206.122817398"
Sep 17 21:09:01.072117 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:09:01.072087    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-k99td"
Sep 17 21:09:02.105419 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:09:02.105367    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:09:02.298226 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:09:02.298187    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:09:06.071993 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:09:06.071950    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-k99td"
Sep 17 21:09:06.084562 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:09:06.084537    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-k99td"
Sep 17 21:09:06.990262 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:09:06.990231    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-k99td"
Sep 17 21:09:09.027722 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:09:09.027688    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:09:12.046914 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:09:12.046878    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:09:12.149533 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:09:12.149495    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:09:12.257607 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:09:12.257552    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:09:17.217826 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:09:17.217791    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:09:17.995969 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:09:17.995941    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-k99td"
Sep 17 21:09:23.022702 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:09:23.022655    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:09:24.037868 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:09:24.037816    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:09:24.198874 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:09:24.198825    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:09:26.986700 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:09:26.986663    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:09:29.972553 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:09:29.972506    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:09:34.415781 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:09:34.415742    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:09:35.927241 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:09:35.927195    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:09:36.023946 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:09:36.023911    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:09:47.161902 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:09:47.161864    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6cfb5b789c-w6tlv"]
Sep 17 21:09:47.168714 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:09:47.168691    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6cfb5b789c-w6tlv"
Sep 17 21:09:47.171568 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:09:47.171544    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Sep 17 21:09:47.171667 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:09:47.171643    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Sep 17 21:09:47.171819 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:09:47.171802    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-226qh\""
Sep 17 21:09:47.172150 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:09:47.172135    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 21:09:47.172195 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:09:47.172134    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Sep 17 21:09:47.172835 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:09:47.172816    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"seaweedfs-s3-creds\""
Sep 17 21:09:47.200244 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:09:47.200216    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6cfb5b789c-w6tlv"]
Sep 17 21:09:47.210511 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:09:47.210465    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:09:47.263106 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:09:47.263074    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94556\" (UniqueName: \"kubernetes.io/projected/3b5e901f-4847-4a79-b26d-c905bdb7657a-kube-api-access-94556\") pod \"auth-enabled-test-kserve-6cfb5b789c-w6tlv\" (UID: \"3b5e901f-4847-4a79-b26d-c905bdb7657a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6cfb5b789c-w6tlv"
Sep 17 21:09:47.263293 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:09:47.263188    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b5e901f-4847-4a79-b26d-c905bdb7657a-model-cache\") pod \"auth-enabled-test-kserve-6cfb5b789c-w6tlv\" (UID: \"3b5e901f-4847-4a79-b26d-c905bdb7657a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6cfb5b789c-w6tlv"
Sep 17 21:09:47.263293 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:09:47.263232    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b5e901f-4847-4a79-b26d-c905bdb7657a-dshm\") pod \"auth-enabled-test-kserve-6cfb5b789c-w6tlv\" (UID: \"3b5e901f-4847-4a79-b26d-c905bdb7657a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6cfb5b789c-w6tlv"
Sep 17 21:09:47.263293 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:09:47.263257    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b5e901f-4847-4a79-b26d-c905bdb7657a-tmp-dir\") pod \"auth-enabled-test-kserve-6cfb5b789c-w6tlv\" (UID: \"3b5e901f-4847-4a79-b26d-c905bdb7657a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6cfb5b789c-w6tlv"
Sep 17 21:09:47.263293 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:09:47.263284    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b5e901f-4847-4a79-b26d-c905bdb7657a-kserve-provision-location\") pod \"auth-enabled-test-kserve-6cfb5b789c-w6tlv\" (UID: \"3b5e901f-4847-4a79-b26d-c905bdb7657a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6cfb5b789c-w6tlv"
Sep 17 21:09:47.263473 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:09:47.263312    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b5e901f-4847-4a79-b26d-c905bdb7657a-cabundle-cert\") pod \"auth-enabled-test-kserve-6cfb5b789c-w6tlv\" (UID: \"3b5e901f-4847-4a79-b26d-c905bdb7657a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6cfb5b789c-w6tlv"
Sep 17 21:09:47.263473 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:09:47.263377    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b5e901f-4847-4a79-b26d-c905bdb7657a-home\") pod \"auth-enabled-test-kserve-6cfb5b789c-w6tlv\" (UID: \"3b5e901f-4847-4a79-b26d-c905bdb7657a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6cfb5b789c-w6tlv"
Sep 17 21:09:47.263473 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:09:47.263401    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b5e901f-4847-4a79-b26d-c905bdb7657a-tls-certs\") pod \"auth-enabled-test-kserve-6cfb5b789c-w6tlv\" (UID: \"3b5e901f-4847-4a79-b26d-c905bdb7657a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6cfb5b789c-w6tlv"
Sep 17 21:09:47.364709 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:09:47.364679    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b5e901f-4847-4a79-b26d-c905bdb7657a-model-cache\") pod \"auth-enabled-test-kserve-6cfb5b789c-w6tlv\" (UID: \"3b5e901f-4847-4a79-b26d-c905bdb7657a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6cfb5b789c-w6tlv"
Sep 17 21:09:47.364900 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:09:47.364723    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b5e901f-4847-4a79-b26d-c905bdb7657a-dshm\") pod \"auth-enabled-test-kserve-6cfb5b789c-w6tlv\" (UID: \"3b5e901f-4847-4a79-b26d-c905bdb7657a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6cfb5b789c-w6tlv"
Sep 17 21:09:47.364900 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:09:47.364745    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b5e901f-4847-4a79-b26d-c905bdb7657a-tmp-dir\") pod \"auth-enabled-test-kserve-6cfb5b789c-w6tlv\" (UID: \"3b5e901f-4847-4a79-b26d-c905bdb7657a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6cfb5b789c-w6tlv"
Sep 17 21:09:47.364900 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:09:47.364770    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b5e901f-4847-4a79-b26d-c905bdb7657a-kserve-provision-location\") pod \"auth-enabled-test-kserve-6cfb5b789c-w6tlv\" (UID: \"3b5e901f-4847-4a79-b26d-c905bdb7657a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6cfb5b789c-w6tlv"
Sep 17 21:09:47.364900 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:09:47.364795    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b5e901f-4847-4a79-b26d-c905bdb7657a-cabundle-cert\") pod \"auth-enabled-test-kserve-6cfb5b789c-w6tlv\" (UID: \"3b5e901f-4847-4a79-b26d-c905bdb7657a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6cfb5b789c-w6tlv"
Sep 17 21:09:47.365138 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:09:47.364901    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b5e901f-4847-4a79-b26d-c905bdb7657a-home\") pod \"auth-enabled-test-kserve-6cfb5b789c-w6tlv\" (UID: \"3b5e901f-4847-4a79-b26d-c905bdb7657a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6cfb5b789c-w6tlv"
Sep 17 21:09:47.365138 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:09:47.364933    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b5e901f-4847-4a79-b26d-c905bdb7657a-tls-certs\") pod \"auth-enabled-test-kserve-6cfb5b789c-w6tlv\" (UID: \"3b5e901f-4847-4a79-b26d-c905bdb7657a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6cfb5b789c-w6tlv"
Sep 17 21:09:47.365138 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:09:47.364984    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94556\" (UniqueName: \"kubernetes.io/projected/3b5e901f-4847-4a79-b26d-c905bdb7657a-kube-api-access-94556\") pod \"auth-enabled-test-kserve-6cfb5b789c-w6tlv\" (UID: \"3b5e901f-4847-4a79-b26d-c905bdb7657a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6cfb5b789c-w6tlv"
Sep 17 21:09:47.365138 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:09:47.365075    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b5e901f-4847-4a79-b26d-c905bdb7657a-model-cache\") pod \"auth-enabled-test-kserve-6cfb5b789c-w6tlv\" (UID: \"3b5e901f-4847-4a79-b26d-c905bdb7657a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6cfb5b789c-w6tlv"
Sep 17 21:09:47.365350 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:09:47.365224    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b5e901f-4847-4a79-b26d-c905bdb7657a-tmp-dir\") pod \"auth-enabled-test-kserve-6cfb5b789c-w6tlv\" (UID: \"3b5e901f-4847-4a79-b26d-c905bdb7657a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6cfb5b789c-w6tlv"
Sep 17 21:09:47.365350 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:09:47.365271    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b5e901f-4847-4a79-b26d-c905bdb7657a-kserve-provision-location\") pod \"auth-enabled-test-kserve-6cfb5b789c-w6tlv\" (UID: \"3b5e901f-4847-4a79-b26d-c905bdb7657a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6cfb5b789c-w6tlv"
Sep 17 21:09:47.365463 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:09:47.365367    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b5e901f-4847-4a79-b26d-c905bdb7657a-home\") pod \"auth-enabled-test-kserve-6cfb5b789c-w6tlv\" (UID: \"3b5e901f-4847-4a79-b26d-c905bdb7657a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6cfb5b789c-w6tlv"
Sep 17 21:09:47.365540 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:09:47.365519    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b5e901f-4847-4a79-b26d-c905bdb7657a-cabundle-cert\") pod \"auth-enabled-test-kserve-6cfb5b789c-w6tlv\" (UID: \"3b5e901f-4847-4a79-b26d-c905bdb7657a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6cfb5b789c-w6tlv"
Sep 17 21:09:47.366991 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:09:47.366972    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b5e901f-4847-4a79-b26d-c905bdb7657a-dshm\") pod \"auth-enabled-test-kserve-6cfb5b789c-w6tlv\" (UID: \"3b5e901f-4847-4a79-b26d-c905bdb7657a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6cfb5b789c-w6tlv"
Sep 17 21:09:47.367445 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:09:47.367428    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b5e901f-4847-4a79-b26d-c905bdb7657a-tls-certs\") pod \"auth-enabled-test-kserve-6cfb5b789c-w6tlv\" (UID: \"3b5e901f-4847-4a79-b26d-c905bdb7657a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6cfb5b789c-w6tlv"
Sep 17 21:09:47.374422 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:09:47.374401    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94556\" (UniqueName: \"kubernetes.io/projected/3b5e901f-4847-4a79-b26d-c905bdb7657a-kube-api-access-94556\") pod \"auth-enabled-test-kserve-6cfb5b789c-w6tlv\" (UID: \"3b5e901f-4847-4a79-b26d-c905bdb7657a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6cfb5b789c-w6tlv"
Sep 17 21:09:47.432937 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:09:47.432832    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:09:47.480898 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:09:47.480866    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6cfb5b789c-w6tlv"
Sep 17 21:09:47.760510 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:09:47.760342    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6cfb5b789c-w6tlv"]
Sep 17 21:09:47.762915 ip-10-0-139-36 kubenswrapper[2575]: W0917 21:09:47.762890    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b5e901f_4847_4a79_b26d_c905bdb7657a.slice/crio-263fadbd4069245ca17cd93d5b1041d46b06e3f90b4b9146b3fa185d187b0aba WatchSource:0}: Error finding container 263fadbd4069245ca17cd93d5b1041d46b06e3f90b4b9146b3fa185d187b0aba: Status 404 returned error can't find the container with id 263fadbd4069245ca17cd93d5b1041d46b06e3f90b4b9146b3fa185d187b0aba
Sep 17 21:09:48.187198 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:09:48.187081    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6cfb5b789c-w6tlv" event={"ID":"3b5e901f-4847-4a79-b26d-c905bdb7657a","Type":"ContainerStarted","Data":"9e3b7b490874d076a117ccd3dcc4e4bc70a55c8cee257cd9e2bac647b6890060"}
Sep 17 21:09:48.187755 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:09:48.187729    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6cfb5b789c-w6tlv" event={"ID":"3b5e901f-4847-4a79-b26d-c905bdb7657a","Type":"ContainerStarted","Data":"263fadbd4069245ca17cd93d5b1041d46b06e3f90b4b9146b3fa185d187b0aba"}
Sep 17 21:09:50.199021 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:09:50.198990    2575 generic.go:356] "Generic (PLEG): container finished" podID="3b5e901f-4847-4a79-b26d-c905bdb7657a" containerID="9e3b7b490874d076a117ccd3dcc4e4bc70a55c8cee257cd9e2bac647b6890060" exitCode=0
Sep 17 21:09:50.199445 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:09:50.199068    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6cfb5b789c-w6tlv" event={"ID":"3b5e901f-4847-4a79-b26d-c905bdb7657a","Type":"ContainerDied","Data":"9e3b7b490874d076a117ccd3dcc4e4bc70a55c8cee257cd9e2bac647b6890060"}
Sep 17 21:09:58.769588 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:09:58.769535    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:10:18.379107 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:10:18.379070    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6cfb5b789c-w6tlv" event={"ID":"3b5e901f-4847-4a79-b26d-c905bdb7657a","Type":"ContainerStarted","Data":"29cd476d103b40e97482cdb1d4700284d7c4cc6e3f02fc02e601ed9d23d857cf"}
Sep 17 21:10:18.403070 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:10:18.403006    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6cfb5b789c-w6tlv" podStartSLOduration=1.935804204 podStartE2EDuration="31.402986139s" totalImagesPullingTime="27.309560353s" totalInitContainerRuntime="2.157621582s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:09:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:10:18.402001228 +0000 UTC m=+1283.554445601" watchObservedRunningTime="2026-09-17 21:10:18.402986139 +0000 UTC m=+1283.555430515"
Sep 17 21:10:18.481287 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:10:18.481256    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6cfb5b789c-w6tlv"
Sep 17 21:10:27.481970 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:10:27.481925    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6cfb5b789c-w6tlv"
Sep 17 21:10:27.483738 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:10:27.483705    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6cfb5b789c-w6tlv" podUID="3b5e901f-4847-4a79-b26d-c905bdb7657a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Sep 17 21:10:37.482490 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:10:37.482449    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6cfb5b789c-w6tlv" podUID="3b5e901f-4847-4a79-b26d-c905bdb7657a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Sep 17 21:10:47.481593 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:10:47.481535    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6cfb5b789c-w6tlv" podUID="3b5e901f-4847-4a79-b26d-c905bdb7657a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Sep 17 21:10:56.480126 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:10:56.480092    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-k99td"]
Sep 17 21:10:56.486616 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:10:56.486579    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:10:56.490759 ip-10-0-139-36 kubenswrapper[2575]: E0917 21:10:56.490722    2575 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 21:10:56.490759 ip-10-0-139-36 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:10:56.490759 ip-10-0-139-36 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 17 21:10:56.490759 ip-10-0-139-36 kubenswrapper[2575]:  > containerID="888a4e2a54eea1d0f3c1d207dee99ea55e0c9c68c3371e47e1be31a2c75bc145" cmd=["/bin/sleep","15"]
Sep 17 21:10:56.491085 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:10:56.490776    2575 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-k99td" message="" err=<
Sep 17 21:10:56.491085 ip-10-0-139-36 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:10:56.491085 ip-10-0-139-36 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 17 21:10:56.491085 ip-10-0-139-36 kubenswrapper[2575]:  >
Sep 17 21:10:56.491085 ip-10-0-139-36 kubenswrapper[2575]: E0917 21:10:56.490788    2575 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 21:10:56.491085 ip-10-0-139-36 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:10:56.491085 ip-10-0-139-36 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 17 21:10:56.491085 ip-10-0-139-36 kubenswrapper[2575]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-k99td" podUID="598ca3a5-3638-4ff3-99a3-118ecd4590d5" containerName="main" containerID="cri-o://888a4e2a54eea1d0f3c1d207dee99ea55e0c9c68c3371e47e1be31a2c75bc145"
Sep 17 21:10:56.491085 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:10:56.490823    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-k99td" podUID="598ca3a5-3638-4ff3-99a3-118ecd4590d5" containerName="main" containerID="cri-o://888a4e2a54eea1d0f3c1d207dee99ea55e0c9c68c3371e47e1be31a2c75bc145" gracePeriod=60
Sep 17 21:10:56.554191 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:10:56.554139    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:10:56.707576 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:10:56.707543    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-k99td" podUID="598ca3a5-3638-4ff3-99a3-118ecd4590d5" containerName="llm-d-routing-sidecar" containerID="cri-o://39505a6bc54656497b841fdd1cd15e6f59ece8239501bf3648e9bed0c28144bf" gracePeriod=60
Sep 17 21:10:56.954944 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:10:56.954917    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-k99td"
Sep 17 21:10:57.055240 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:10:57.055203    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/598ca3a5-3638-4ff3-99a3-118ecd4590d5-dshm\") pod \"598ca3a5-3638-4ff3-99a3-118ecd4590d5\" (UID: \"598ca3a5-3638-4ff3-99a3-118ecd4590d5\") "
Sep 17 21:10:57.055425 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:10:57.055249    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/598ca3a5-3638-4ff3-99a3-118ecd4590d5-model-cache\") pod \"598ca3a5-3638-4ff3-99a3-118ecd4590d5\" (UID: \"598ca3a5-3638-4ff3-99a3-118ecd4590d5\") "
Sep 17 21:10:57.055425 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:10:57.055278    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pgt9m\" (UniqueName: \"kubernetes.io/projected/598ca3a5-3638-4ff3-99a3-118ecd4590d5-kube-api-access-pgt9m\") pod \"598ca3a5-3638-4ff3-99a3-118ecd4590d5\" (UID: \"598ca3a5-3638-4ff3-99a3-118ecd4590d5\") "
Sep 17 21:10:57.055425 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:10:57.055367    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/598ca3a5-3638-4ff3-99a3-118ecd4590d5-tmp-dir\") pod \"598ca3a5-3638-4ff3-99a3-118ecd4590d5\" (UID: \"598ca3a5-3638-4ff3-99a3-118ecd4590d5\") "
Sep 17 21:10:57.055600 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:10:57.055458    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/598ca3a5-3638-4ff3-99a3-118ecd4590d5-home\") pod \"598ca3a5-3638-4ff3-99a3-118ecd4590d5\" (UID: \"598ca3a5-3638-4ff3-99a3-118ecd4590d5\") "
Sep 17 21:10:57.055600 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:10:57.055479    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/598ca3a5-3638-4ff3-99a3-118ecd4590d5-model-cache" (OuterVolumeSpecName: "model-cache") pod "598ca3a5-3638-4ff3-99a3-118ecd4590d5" (UID: "598ca3a5-3638-4ff3-99a3-118ecd4590d5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:10:57.055600 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:10:57.055534    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/598ca3a5-3638-4ff3-99a3-118ecd4590d5-tls-certs\") pod \"598ca3a5-3638-4ff3-99a3-118ecd4590d5\" (UID: \"598ca3a5-3638-4ff3-99a3-118ecd4590d5\") "
Sep 17 21:10:57.055600 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:10:57.055573    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/598ca3a5-3638-4ff3-99a3-118ecd4590d5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "598ca3a5-3638-4ff3-99a3-118ecd4590d5" (UID: "598ca3a5-3638-4ff3-99a3-118ecd4590d5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:10:57.055833 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:10:57.055635    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/598ca3a5-3638-4ff3-99a3-118ecd4590d5-home" (OuterVolumeSpecName: "home") pod "598ca3a5-3638-4ff3-99a3-118ecd4590d5" (UID: "598ca3a5-3638-4ff3-99a3-118ecd4590d5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:10:57.055965 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:10:57.055942    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/598ca3a5-3638-4ff3-99a3-118ecd4590d5-model-cache\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:10:57.056089 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:10:57.055971    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/598ca3a5-3638-4ff3-99a3-118ecd4590d5-tmp-dir\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:10:57.056089 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:10:57.055984    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/598ca3a5-3638-4ff3-99a3-118ecd4590d5-home\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:10:57.057625 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:10:57.057585    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/598ca3a5-3638-4ff3-99a3-118ecd4590d5-dshm" (OuterVolumeSpecName: "dshm") pod "598ca3a5-3638-4ff3-99a3-118ecd4590d5" (UID: "598ca3a5-3638-4ff3-99a3-118ecd4590d5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:10:57.057775 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:10:57.057749    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/598ca3a5-3638-4ff3-99a3-118ecd4590d5-kube-api-access-pgt9m" (OuterVolumeSpecName: "kube-api-access-pgt9m") pod "598ca3a5-3638-4ff3-99a3-118ecd4590d5" (UID: "598ca3a5-3638-4ff3-99a3-118ecd4590d5"). InnerVolumeSpecName "kube-api-access-pgt9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:10:57.058027 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:10:57.057998    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/598ca3a5-3638-4ff3-99a3-118ecd4590d5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "598ca3a5-3638-4ff3-99a3-118ecd4590d5" (UID: "598ca3a5-3638-4ff3-99a3-118ecd4590d5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:10:57.157029 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:10:57.156843    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/598ca3a5-3638-4ff3-99a3-118ecd4590d5-tls-certs\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:10:57.157029 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:10:57.156920    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/598ca3a5-3638-4ff3-99a3-118ecd4590d5-dshm\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:10:57.157029 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:10:57.156935    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgt9m\" (UniqueName: \"kubernetes.io/projected/598ca3a5-3638-4ff3-99a3-118ecd4590d5-kube-api-access-pgt9m\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:10:57.481780 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:10:57.481741    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6cfb5b789c-w6tlv" podUID="3b5e901f-4847-4a79-b26d-c905bdb7657a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Sep 17 21:10:57.590358 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:10:57.590323    2575 generic.go:356] "Generic (PLEG): container finished" podID="598ca3a5-3638-4ff3-99a3-118ecd4590d5" containerID="888a4e2a54eea1d0f3c1d207dee99ea55e0c9c68c3371e47e1be31a2c75bc145" exitCode=0
Sep 17 21:10:57.590358 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:10:57.590349    2575 generic.go:356] "Generic (PLEG): container finished" podID="598ca3a5-3638-4ff3-99a3-118ecd4590d5" containerID="39505a6bc54656497b841fdd1cd15e6f59ece8239501bf3648e9bed0c28144bf" exitCode=0
Sep 17 21:10:57.590608 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:10:57.590386    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-k99td"
Sep 17 21:10:57.590608 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:10:57.590413    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-k99td" event={"ID":"598ca3a5-3638-4ff3-99a3-118ecd4590d5","Type":"ContainerDied","Data":"888a4e2a54eea1d0f3c1d207dee99ea55e0c9c68c3371e47e1be31a2c75bc145"}
Sep 17 21:10:57.590608 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:10:57.590473    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-k99td" event={"ID":"598ca3a5-3638-4ff3-99a3-118ecd4590d5","Type":"ContainerDied","Data":"39505a6bc54656497b841fdd1cd15e6f59ece8239501bf3648e9bed0c28144bf"}
Sep 17 21:10:57.590608 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:10:57.590492    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-k99td" event={"ID":"598ca3a5-3638-4ff3-99a3-118ecd4590d5","Type":"ContainerDied","Data":"dfdfa6a1be934d8eb81eb2ad821fab21d7376691e7c7072214373b69684a84a0"}
Sep 17 21:10:57.590608 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:10:57.590513    2575 scope.go:117] "RemoveContainer" containerID="888a4e2a54eea1d0f3c1d207dee99ea55e0c9c68c3371e47e1be31a2c75bc145"
Sep 17 21:10:57.599499 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:10:57.599472    2575 scope.go:117] "RemoveContainer" containerID="39505a6bc54656497b841fdd1cd15e6f59ece8239501bf3648e9bed0c28144bf"
Sep 17 21:10:57.606387 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:10:57.606367    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-k99td"]
Sep 17 21:10:57.610120 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:10:57.610100    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-k99td"]
Sep 17 21:10:57.612149 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:10:57.612134    2575 scope.go:117] "RemoveContainer" containerID="888a4e2a54eea1d0f3c1d207dee99ea55e0c9c68c3371e47e1be31a2c75bc145"
Sep 17 21:10:57.612429 ip-10-0-139-36 kubenswrapper[2575]: E0917 21:10:57.612407    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"888a4e2a54eea1d0f3c1d207dee99ea55e0c9c68c3371e47e1be31a2c75bc145\": container with ID starting with 888a4e2a54eea1d0f3c1d207dee99ea55e0c9c68c3371e47e1be31a2c75bc145 not found: ID does not exist" containerID="888a4e2a54eea1d0f3c1d207dee99ea55e0c9c68c3371e47e1be31a2c75bc145"
Sep 17 21:10:57.612501 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:10:57.612438    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"888a4e2a54eea1d0f3c1d207dee99ea55e0c9c68c3371e47e1be31a2c75bc145"} err="failed to get container status \"888a4e2a54eea1d0f3c1d207dee99ea55e0c9c68c3371e47e1be31a2c75bc145\": rpc error: code = NotFound desc = could not find container \"888a4e2a54eea1d0f3c1d207dee99ea55e0c9c68c3371e47e1be31a2c75bc145\": container with ID starting with 888a4e2a54eea1d0f3c1d207dee99ea55e0c9c68c3371e47e1be31a2c75bc145 not found: ID does not exist"
Sep 17 21:10:57.612501 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:10:57.612456    2575 scope.go:117] "RemoveContainer" containerID="39505a6bc54656497b841fdd1cd15e6f59ece8239501bf3648e9bed0c28144bf"
Sep 17 21:10:57.612760 ip-10-0-139-36 kubenswrapper[2575]: E0917 21:10:57.612746    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39505a6bc54656497b841fdd1cd15e6f59ece8239501bf3648e9bed0c28144bf\": container with ID starting with 39505a6bc54656497b841fdd1cd15e6f59ece8239501bf3648e9bed0c28144bf not found: ID does not exist" containerID="39505a6bc54656497b841fdd1cd15e6f59ece8239501bf3648e9bed0c28144bf"
Sep 17 21:10:57.612804 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:10:57.612763    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39505a6bc54656497b841fdd1cd15e6f59ece8239501bf3648e9bed0c28144bf"} err="failed to get container status \"39505a6bc54656497b841fdd1cd15e6f59ece8239501bf3648e9bed0c28144bf\": rpc error: code = NotFound desc = could not find container \"39505a6bc54656497b841fdd1cd15e6f59ece8239501bf3648e9bed0c28144bf\": container with ID starting with 39505a6bc54656497b841fdd1cd15e6f59ece8239501bf3648e9bed0c28144bf not found: ID does not exist"
Sep 17 21:10:57.612804 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:10:57.612775    2575 scope.go:117] "RemoveContainer" containerID="888a4e2a54eea1d0f3c1d207dee99ea55e0c9c68c3371e47e1be31a2c75bc145"
Sep 17 21:10:57.613062 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:10:57.613038    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"888a4e2a54eea1d0f3c1d207dee99ea55e0c9c68c3371e47e1be31a2c75bc145"} err="failed to get container status \"888a4e2a54eea1d0f3c1d207dee99ea55e0c9c68c3371e47e1be31a2c75bc145\": rpc error: code = NotFound desc = could not find container \"888a4e2a54eea1d0f3c1d207dee99ea55e0c9c68c3371e47e1be31a2c75bc145\": container with ID starting with 888a4e2a54eea1d0f3c1d207dee99ea55e0c9c68c3371e47e1be31a2c75bc145 not found: ID does not exist"
Sep 17 21:10:57.613118 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:10:57.613062    2575 scope.go:117] "RemoveContainer" containerID="39505a6bc54656497b841fdd1cd15e6f59ece8239501bf3648e9bed0c28144bf"
Sep 17 21:10:57.613289 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:10:57.613268    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39505a6bc54656497b841fdd1cd15e6f59ece8239501bf3648e9bed0c28144bf"} err="failed to get container status \"39505a6bc54656497b841fdd1cd15e6f59ece8239501bf3648e9bed0c28144bf\": rpc error: code = NotFound desc = could not find container \"39505a6bc54656497b841fdd1cd15e6f59ece8239501bf3648e9bed0c28144bf\": container with ID starting with 39505a6bc54656497b841fdd1cd15e6f59ece8239501bf3648e9bed0c28144bf not found: ID does not exist"
Sep 17 21:10:59.357956 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:10:59.357923    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="598ca3a5-3638-4ff3-99a3-118ecd4590d5" path="/var/lib/kubelet/pods/598ca3a5-3638-4ff3-99a3-118ecd4590d5/volumes"
Sep 17 21:11:07.481722 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:11:07.481677    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6cfb5b789c-w6tlv" podUID="3b5e901f-4847-4a79-b26d-c905bdb7657a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Sep 17 21:11:17.481766 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:11:17.481715    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6cfb5b789c-w6tlv" podUID="3b5e901f-4847-4a79-b26d-c905bdb7657a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Sep 17 21:11:27.482210 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:11:27.482163    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6cfb5b789c-w6tlv" podUID="3b5e901f-4847-4a79-b26d-c905bdb7657a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Sep 17 21:11:29.534880 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:11:29.534830    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:11:29.595724 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:11:29.595686    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:11:37.481468 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:11:37.481418    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6cfb5b789c-w6tlv" podUID="3b5e901f-4847-4a79-b26d-c905bdb7657a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Sep 17 21:11:46.507469 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:11:46.507430    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:11:47.482013 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:11:47.481964    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6cfb5b789c-w6tlv" podUID="3b5e901f-4847-4a79-b26d-c905bdb7657a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Sep 17 21:11:48.467304 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:11:48.467259    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:11:52.938888 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:11:52.938803    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:11:57.482120 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:11:57.482079    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6cfb5b789c-w6tlv" podUID="3b5e901f-4847-4a79-b26d-c905bdb7657a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Sep 17 21:11:58.711240 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:11:58.711206    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:11:58.783226 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:11:58.783153    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:12:05.597217 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:12:05.597174    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:12:05.822720 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:12:05.822682    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:12:07.482306 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:12:07.482258    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6cfb5b789c-w6tlv" podUID="3b5e901f-4847-4a79-b26d-c905bdb7657a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Sep 17 21:12:17.492402 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:12:17.492368    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6cfb5b789c-w6tlv"
Sep 17 21:12:17.500217 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:12:17.500180    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6cfb5b789c-w6tlv"
Sep 17 21:12:21.852427 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:12:21.852392    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:12:46.042099 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:12:46.042058    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:12:50.291564 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:12:50.291356    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:12:50.292952 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:12:50.292923    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6cfb5b789c-w6tlv"]
Sep 17 21:12:50.369322 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:12:50.369285    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:12:53.253322 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:12:53.253283    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:12:53.328153 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:12:53.328124    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:13:05.315586 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:05.315540    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6cfb5b789c-w6tlv" podUID="3b5e901f-4847-4a79-b26d-c905bdb7657a" containerName="main" containerID="cri-o://29cd476d103b40e97482cdb1d4700284d7c4cc6e3f02fc02e601ed9d23d857cf" gracePeriod=45
Sep 17 21:13:05.493725 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:05.493679    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6cfb5b789c-w6tlv" podUID="3b5e901f-4847-4a79-b26d-c905bdb7657a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Sep 17 21:13:06.070283 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:06.070219    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6cfb5b789c-w6tlv"
Sep 17 21:13:06.243361 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:06.243318    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b5e901f-4847-4a79-b26d-c905bdb7657a-home\") pod \"3b5e901f-4847-4a79-b26d-c905bdb7657a\" (UID: \"3b5e901f-4847-4a79-b26d-c905bdb7657a\") "
Sep 17 21:13:06.243573 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:06.243387    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b5e901f-4847-4a79-b26d-c905bdb7657a-dshm\") pod \"3b5e901f-4847-4a79-b26d-c905bdb7657a\" (UID: \"3b5e901f-4847-4a79-b26d-c905bdb7657a\") "
Sep 17 21:13:06.243573 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:06.243444    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b5e901f-4847-4a79-b26d-c905bdb7657a-tmp-dir\") pod \"3b5e901f-4847-4a79-b26d-c905bdb7657a\" (UID: \"3b5e901f-4847-4a79-b26d-c905bdb7657a\") "
Sep 17 21:13:06.243573 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:06.243482    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b5e901f-4847-4a79-b26d-c905bdb7657a-cabundle-cert\") pod \"3b5e901f-4847-4a79-b26d-c905bdb7657a\" (UID: \"3b5e901f-4847-4a79-b26d-c905bdb7657a\") "
Sep 17 21:13:06.243573 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:06.243516    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b5e901f-4847-4a79-b26d-c905bdb7657a-tls-certs\") pod \"3b5e901f-4847-4a79-b26d-c905bdb7657a\" (UID: \"3b5e901f-4847-4a79-b26d-c905bdb7657a\") "
Sep 17 21:13:06.243573 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:06.243559    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b5e901f-4847-4a79-b26d-c905bdb7657a-kserve-provision-location\") pod \"3b5e901f-4847-4a79-b26d-c905bdb7657a\" (UID: \"3b5e901f-4847-4a79-b26d-c905bdb7657a\") "
Sep 17 21:13:06.243888 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:06.243600    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b5e901f-4847-4a79-b26d-c905bdb7657a-model-cache\") pod \"3b5e901f-4847-4a79-b26d-c905bdb7657a\" (UID: \"3b5e901f-4847-4a79-b26d-c905bdb7657a\") "
Sep 17 21:13:06.243888 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:06.243637    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-94556\" (UniqueName: \"kubernetes.io/projected/3b5e901f-4847-4a79-b26d-c905bdb7657a-kube-api-access-94556\") pod \"3b5e901f-4847-4a79-b26d-c905bdb7657a\" (UID: \"3b5e901f-4847-4a79-b26d-c905bdb7657a\") "
Sep 17 21:13:06.244008 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:06.243881    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b5e901f-4847-4a79-b26d-c905bdb7657a-model-cache" (OuterVolumeSpecName: "model-cache") pod "3b5e901f-4847-4a79-b26d-c905bdb7657a" (UID: "3b5e901f-4847-4a79-b26d-c905bdb7657a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:13:06.244060 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:06.244005    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b5e901f-4847-4a79-b26d-c905bdb7657a-model-cache\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:13:06.244291 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:06.244253    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b5e901f-4847-4a79-b26d-c905bdb7657a-home" (OuterVolumeSpecName: "home") pod "3b5e901f-4847-4a79-b26d-c905bdb7657a" (UID: "3b5e901f-4847-4a79-b26d-c905bdb7657a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:13:06.244436 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:06.244372    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b5e901f-4847-4a79-b26d-c905bdb7657a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3b5e901f-4847-4a79-b26d-c905bdb7657a" (UID: "3b5e901f-4847-4a79-b26d-c905bdb7657a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:13:06.245879 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:06.245837    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b5e901f-4847-4a79-b26d-c905bdb7657a-dshm" (OuterVolumeSpecName: "dshm") pod "3b5e901f-4847-4a79-b26d-c905bdb7657a" (UID: "3b5e901f-4847-4a79-b26d-c905bdb7657a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:13:06.246162 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:06.246128    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b5e901f-4847-4a79-b26d-c905bdb7657a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3b5e901f-4847-4a79-b26d-c905bdb7657a" (UID: "3b5e901f-4847-4a79-b26d-c905bdb7657a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:13:06.246329 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:06.246312    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b5e901f-4847-4a79-b26d-c905bdb7657a-kube-api-access-94556" (OuterVolumeSpecName: "kube-api-access-94556") pod "3b5e901f-4847-4a79-b26d-c905bdb7657a" (UID: "3b5e901f-4847-4a79-b26d-c905bdb7657a"). InnerVolumeSpecName "kube-api-access-94556". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:13:06.258989 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:06.258962    2575 generic.go:356] "Generic (PLEG): container finished" podID="3b5e901f-4847-4a79-b26d-c905bdb7657a" containerID="29cd476d103b40e97482cdb1d4700284d7c4cc6e3f02fc02e601ed9d23d857cf" exitCode=0
Sep 17 21:13:06.259115 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:06.259016    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6cfb5b789c-w6tlv"
Sep 17 21:13:06.259115 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:06.259042    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6cfb5b789c-w6tlv" event={"ID":"3b5e901f-4847-4a79-b26d-c905bdb7657a","Type":"ContainerDied","Data":"29cd476d103b40e97482cdb1d4700284d7c4cc6e3f02fc02e601ed9d23d857cf"}
Sep 17 21:13:06.259115 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:06.259096    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6cfb5b789c-w6tlv" event={"ID":"3b5e901f-4847-4a79-b26d-c905bdb7657a","Type":"ContainerDied","Data":"263fadbd4069245ca17cd93d5b1041d46b06e3f90b4b9146b3fa185d187b0aba"}
Sep 17 21:13:06.259312 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:06.259118    2575 scope.go:117] "RemoveContainer" containerID="29cd476d103b40e97482cdb1d4700284d7c4cc6e3f02fc02e601ed9d23d857cf"
Sep 17 21:13:06.262098 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:06.262070    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b5e901f-4847-4a79-b26d-c905bdb7657a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3b5e901f-4847-4a79-b26d-c905bdb7657a" (UID: "3b5e901f-4847-4a79-b26d-c905bdb7657a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:13:06.269431 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:06.269414    2575 scope.go:117] "RemoveContainer" containerID="9e3b7b490874d076a117ccd3dcc4e4bc70a55c8cee257cd9e2bac647b6890060"
Sep 17 21:13:06.272340 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:06.272316    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b5e901f-4847-4a79-b26d-c905bdb7657a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3b5e901f-4847-4a79-b26d-c905bdb7657a" (UID: "3b5e901f-4847-4a79-b26d-c905bdb7657a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:13:06.277390 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:06.277368    2575 scope.go:117] "RemoveContainer" containerID="29cd476d103b40e97482cdb1d4700284d7c4cc6e3f02fc02e601ed9d23d857cf"
Sep 17 21:13:06.277654 ip-10-0-139-36 kubenswrapper[2575]: E0917 21:13:06.277630    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29cd476d103b40e97482cdb1d4700284d7c4cc6e3f02fc02e601ed9d23d857cf\": container with ID starting with 29cd476d103b40e97482cdb1d4700284d7c4cc6e3f02fc02e601ed9d23d857cf not found: ID does not exist" containerID="29cd476d103b40e97482cdb1d4700284d7c4cc6e3f02fc02e601ed9d23d857cf"
Sep 17 21:13:06.277713 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:06.277662    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29cd476d103b40e97482cdb1d4700284d7c4cc6e3f02fc02e601ed9d23d857cf"} err="failed to get container status \"29cd476d103b40e97482cdb1d4700284d7c4cc6e3f02fc02e601ed9d23d857cf\": rpc error: code = NotFound desc = could not find container \"29cd476d103b40e97482cdb1d4700284d7c4cc6e3f02fc02e601ed9d23d857cf\": container with ID starting with 29cd476d103b40e97482cdb1d4700284d7c4cc6e3f02fc02e601ed9d23d857cf not found: ID does not exist"
Sep 17 21:13:06.277713 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:06.277682    2575 scope.go:117] "RemoveContainer" containerID="9e3b7b490874d076a117ccd3dcc4e4bc70a55c8cee257cd9e2bac647b6890060"
Sep 17 21:13:06.277950 ip-10-0-139-36 kubenswrapper[2575]: E0917 21:13:06.277924    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e3b7b490874d076a117ccd3dcc4e4bc70a55c8cee257cd9e2bac647b6890060\": container with ID starting with 9e3b7b490874d076a117ccd3dcc4e4bc70a55c8cee257cd9e2bac647b6890060 not found: ID does not exist" containerID="9e3b7b490874d076a117ccd3dcc4e4bc70a55c8cee257cd9e2bac647b6890060"
Sep 17 21:13:06.278031 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:06.277954    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e3b7b490874d076a117ccd3dcc4e4bc70a55c8cee257cd9e2bac647b6890060"} err="failed to get container status \"9e3b7b490874d076a117ccd3dcc4e4bc70a55c8cee257cd9e2bac647b6890060\": rpc error: code = NotFound desc = could not find container \"9e3b7b490874d076a117ccd3dcc4e4bc70a55c8cee257cd9e2bac647b6890060\": container with ID starting with 9e3b7b490874d076a117ccd3dcc4e4bc70a55c8cee257cd9e2bac647b6890060 not found: ID does not exist"
Sep 17 21:13:06.345255 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:06.345178    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b5e901f-4847-4a79-b26d-c905bdb7657a-kserve-provision-location\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:13:06.345255 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:06.345209    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94556\" (UniqueName: \"kubernetes.io/projected/3b5e901f-4847-4a79-b26d-c905bdb7657a-kube-api-access-94556\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:13:06.345255 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:06.345219    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b5e901f-4847-4a79-b26d-c905bdb7657a-home\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:13:06.345255 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:06.345226    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b5e901f-4847-4a79-b26d-c905bdb7657a-dshm\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:13:06.345255 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:06.345234    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b5e901f-4847-4a79-b26d-c905bdb7657a-tmp-dir\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:13:06.345255 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:06.345243    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b5e901f-4847-4a79-b26d-c905bdb7657a-cabundle-cert\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:13:06.345255 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:06.345256    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b5e901f-4847-4a79-b26d-c905bdb7657a-tls-certs\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:13:06.584737 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:06.582836    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6cfb5b789c-w6tlv"]
Sep 17 21:13:06.590320 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:06.590291    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6cfb5b789c-w6tlv"]
Sep 17 21:13:07.357300 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:07.357257    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b5e901f-4847-4a79-b26d-c905bdb7657a" path="/var/lib/kubelet/pods/3b5e901f-4847-4a79-b26d-c905bdb7657a/volumes"
Sep 17 21:13:38.851917 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:38.851877    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:13:38.977420 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:38.977378    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:13:42.526685 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:42.526636    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff44d446f-582ts"]
Sep 17 21:13:42.527179 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:42.527104    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b5e901f-4847-4a79-b26d-c905bdb7657a" containerName="storage-initializer"
Sep 17 21:13:42.527179 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:42.527123    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b5e901f-4847-4a79-b26d-c905bdb7657a" containerName="storage-initializer"
Sep 17 21:13:42.527179 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:42.527144    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b5e901f-4847-4a79-b26d-c905bdb7657a" containerName="main"
Sep 17 21:13:42.527179 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:42.527152    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b5e901f-4847-4a79-b26d-c905bdb7657a" containerName="main"
Sep 17 21:13:42.527179 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:42.527164    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="598ca3a5-3638-4ff3-99a3-118ecd4590d5" containerName="llm-d-routing-sidecar"
Sep 17 21:13:42.527179 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:42.527180    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="598ca3a5-3638-4ff3-99a3-118ecd4590d5" containerName="llm-d-routing-sidecar"
Sep 17 21:13:42.527433 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:42.527190    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="598ca3a5-3638-4ff3-99a3-118ecd4590d5" containerName="main"
Sep 17 21:13:42.527433 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:42.527198    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="598ca3a5-3638-4ff3-99a3-118ecd4590d5" containerName="main"
Sep 17 21:13:42.527433 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:42.527282    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b5e901f-4847-4a79-b26d-c905bdb7657a" containerName="main"
Sep 17 21:13:42.527433 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:42.527292    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="598ca3a5-3638-4ff3-99a3-118ecd4590d5" containerName="llm-d-routing-sidecar"
Sep 17 21:13:42.527433 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:42.527302    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="598ca3a5-3638-4ff3-99a3-118ecd4590d5" containerName="main"
Sep 17 21:13:42.530446 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:42.530426    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff44d446f-582ts"
Sep 17 21:13:42.532947 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:42.532924    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Sep 17 21:13:42.533107 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:42.533086    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"seaweedfs-s3-creds\""
Sep 17 21:13:42.533177 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:42.533102    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 21:13:42.533177 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:42.533086    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Sep 17 21:13:42.533177 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:42.533149    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-grsbv\""
Sep 17 21:13:42.533642 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:42.533626    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Sep 17 21:13:42.542888 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:42.542867    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff44d446f-582ts"]
Sep 17 21:13:42.553914 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:42.553882    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:13:42.650875 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:42.650821    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66d48507-65f6-4324-ae1b-0f00cd994c79-home\") pod \"auth-invalid-token-test-kserve-5ff44d446f-582ts\" (UID: \"66d48507-65f6-4324-ae1b-0f00cd994c79\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff44d446f-582ts"
Sep 17 21:13:42.651058 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:42.650927    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66d48507-65f6-4324-ae1b-0f00cd994c79-dshm\") pod \"auth-invalid-token-test-kserve-5ff44d446f-582ts\" (UID: \"66d48507-65f6-4324-ae1b-0f00cd994c79\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff44d446f-582ts"
Sep 17 21:13:42.651058 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:42.650958    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66d48507-65f6-4324-ae1b-0f00cd994c79-cabundle-cert\") pod \"auth-invalid-token-test-kserve-5ff44d446f-582ts\" (UID: \"66d48507-65f6-4324-ae1b-0f00cd994c79\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff44d446f-582ts"
Sep 17 21:13:42.651058 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:42.650982    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66d48507-65f6-4324-ae1b-0f00cd994c79-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-5ff44d446f-582ts\" (UID: \"66d48507-65f6-4324-ae1b-0f00cd994c79\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff44d446f-582ts"
Sep 17 21:13:42.651058 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:42.651005    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66d48507-65f6-4324-ae1b-0f00cd994c79-tls-certs\") pod \"auth-invalid-token-test-kserve-5ff44d446f-582ts\" (UID: \"66d48507-65f6-4324-ae1b-0f00cd994c79\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff44d446f-582ts"
Sep 17 21:13:42.651267 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:42.651083    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66d48507-65f6-4324-ae1b-0f00cd994c79-model-cache\") pod \"auth-invalid-token-test-kserve-5ff44d446f-582ts\" (UID: \"66d48507-65f6-4324-ae1b-0f00cd994c79\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff44d446f-582ts"
Sep 17 21:13:42.651267 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:42.651134    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66d48507-65f6-4324-ae1b-0f00cd994c79-tmp-dir\") pod \"auth-invalid-token-test-kserve-5ff44d446f-582ts\" (UID: \"66d48507-65f6-4324-ae1b-0f00cd994c79\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff44d446f-582ts"
Sep 17 21:13:42.651267 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:42.651157    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg54p\" (UniqueName: \"kubernetes.io/projected/66d48507-65f6-4324-ae1b-0f00cd994c79-kube-api-access-xg54p\") pod \"auth-invalid-token-test-kserve-5ff44d446f-582ts\" (UID: \"66d48507-65f6-4324-ae1b-0f00cd994c79\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff44d446f-582ts"
Sep 17 21:13:42.752228 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:42.752195    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66d48507-65f6-4324-ae1b-0f00cd994c79-home\") pod \"auth-invalid-token-test-kserve-5ff44d446f-582ts\" (UID: \"66d48507-65f6-4324-ae1b-0f00cd994c79\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff44d446f-582ts"
Sep 17 21:13:42.752390 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:42.752238    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66d48507-65f6-4324-ae1b-0f00cd994c79-dshm\") pod \"auth-invalid-token-test-kserve-5ff44d446f-582ts\" (UID: \"66d48507-65f6-4324-ae1b-0f00cd994c79\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff44d446f-582ts"
Sep 17 21:13:42.752390 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:42.752255    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66d48507-65f6-4324-ae1b-0f00cd994c79-cabundle-cert\") pod \"auth-invalid-token-test-kserve-5ff44d446f-582ts\" (UID: \"66d48507-65f6-4324-ae1b-0f00cd994c79\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff44d446f-582ts"
Sep 17 21:13:42.752390 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:42.752277    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66d48507-65f6-4324-ae1b-0f00cd994c79-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-5ff44d446f-582ts\" (UID: \"66d48507-65f6-4324-ae1b-0f00cd994c79\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff44d446f-582ts"
Sep 17 21:13:42.752390 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:42.752297    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66d48507-65f6-4324-ae1b-0f00cd994c79-tls-certs\") pod \"auth-invalid-token-test-kserve-5ff44d446f-582ts\" (UID: \"66d48507-65f6-4324-ae1b-0f00cd994c79\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff44d446f-582ts"
Sep 17 21:13:42.752390 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:42.752326    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66d48507-65f6-4324-ae1b-0f00cd994c79-model-cache\") pod \"auth-invalid-token-test-kserve-5ff44d446f-582ts\" (UID: \"66d48507-65f6-4324-ae1b-0f00cd994c79\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff44d446f-582ts"
Sep 17 21:13:42.752390 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:42.752356    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66d48507-65f6-4324-ae1b-0f00cd994c79-tmp-dir\") pod \"auth-invalid-token-test-kserve-5ff44d446f-582ts\" (UID: \"66d48507-65f6-4324-ae1b-0f00cd994c79\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff44d446f-582ts"
Sep 17 21:13:42.752390 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:42.752380    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg54p\" (UniqueName: \"kubernetes.io/projected/66d48507-65f6-4324-ae1b-0f00cd994c79-kube-api-access-xg54p\") pod \"auth-invalid-token-test-kserve-5ff44d446f-582ts\" (UID: \"66d48507-65f6-4324-ae1b-0f00cd994c79\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff44d446f-582ts"
Sep 17 21:13:42.752784 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:42.752748    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66d48507-65f6-4324-ae1b-0f00cd994c79-home\") pod \"auth-invalid-token-test-kserve-5ff44d446f-582ts\" (UID: \"66d48507-65f6-4324-ae1b-0f00cd994c79\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff44d446f-582ts"
Sep 17 21:13:42.752905 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:42.752880    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66d48507-65f6-4324-ae1b-0f00cd994c79-model-cache\") pod \"auth-invalid-token-test-kserve-5ff44d446f-582ts\" (UID: \"66d48507-65f6-4324-ae1b-0f00cd994c79\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff44d446f-582ts"
Sep 17 21:13:42.752978 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:42.752926    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66d48507-65f6-4324-ae1b-0f00cd994c79-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-5ff44d446f-582ts\" (UID: \"66d48507-65f6-4324-ae1b-0f00cd994c79\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff44d446f-582ts"
Sep 17 21:13:42.752978 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:42.752957    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66d48507-65f6-4324-ae1b-0f00cd994c79-tmp-dir\") pod \"auth-invalid-token-test-kserve-5ff44d446f-582ts\" (UID: \"66d48507-65f6-4324-ae1b-0f00cd994c79\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff44d446f-582ts"
Sep 17 21:13:42.753093 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:42.753076    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66d48507-65f6-4324-ae1b-0f00cd994c79-cabundle-cert\") pod \"auth-invalid-token-test-kserve-5ff44d446f-582ts\" (UID: \"66d48507-65f6-4324-ae1b-0f00cd994c79\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff44d446f-582ts"
Sep 17 21:13:42.754528 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:42.754509    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66d48507-65f6-4324-ae1b-0f00cd994c79-dshm\") pod \"auth-invalid-token-test-kserve-5ff44d446f-582ts\" (UID: \"66d48507-65f6-4324-ae1b-0f00cd994c79\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff44d446f-582ts"
Sep 17 21:13:42.754839 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:42.754823    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66d48507-65f6-4324-ae1b-0f00cd994c79-tls-certs\") pod \"auth-invalid-token-test-kserve-5ff44d446f-582ts\" (UID: \"66d48507-65f6-4324-ae1b-0f00cd994c79\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff44d446f-582ts"
Sep 17 21:13:42.763013 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:42.762988    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg54p\" (UniqueName: \"kubernetes.io/projected/66d48507-65f6-4324-ae1b-0f00cd994c79-kube-api-access-xg54p\") pod \"auth-invalid-token-test-kserve-5ff44d446f-582ts\" (UID: \"66d48507-65f6-4324-ae1b-0f00cd994c79\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff44d446f-582ts"
Sep 17 21:13:42.805703 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:42.805619    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:13:42.841601 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:42.841572    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff44d446f-582ts"
Sep 17 21:13:42.895698 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:42.895502    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:13:43.010925 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:43.010884    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:13:43.119858 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:43.119805    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff44d446f-582ts"]
Sep 17 21:13:43.121518 ip-10-0-139-36 kubenswrapper[2575]: W0917 21:13:43.121493    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66d48507_65f6_4324_ae1b_0f00cd994c79.slice/crio-5e81f99c0b9a9543ed15b98c7d983319cc82845ee139ca3e44288f8ecfa4abaf WatchSource:0}: Error finding container 5e81f99c0b9a9543ed15b98c7d983319cc82845ee139ca3e44288f8ecfa4abaf: Status 404 returned error can't find the container with id 5e81f99c0b9a9543ed15b98c7d983319cc82845ee139ca3e44288f8ecfa4abaf
Sep 17 21:13:43.437590 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:43.437495    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff44d446f-582ts" event={"ID":"66d48507-65f6-4324-ae1b-0f00cd994c79","Type":"ContainerStarted","Data":"8e7fe9e02f3a7ee203f34006c16df36be582063a4852152cedb20e42bfd7f44c"}
Sep 17 21:13:43.437590 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:43.437562    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff44d446f-582ts" event={"ID":"66d48507-65f6-4324-ae1b-0f00cd994c79","Type":"ContainerStarted","Data":"5e81f99c0b9a9543ed15b98c7d983319cc82845ee139ca3e44288f8ecfa4abaf"}
Sep 17 21:13:45.984109 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:45.984074    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:13:46.456085 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:46.456051    2575 generic.go:356] "Generic (PLEG): container finished" podID="66d48507-65f6-4324-ae1b-0f00cd994c79" containerID="8e7fe9e02f3a7ee203f34006c16df36be582063a4852152cedb20e42bfd7f44c" exitCode=0
Sep 17 21:13:46.456258 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:46.456126    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff44d446f-582ts" event={"ID":"66d48507-65f6-4324-ae1b-0f00cd994c79","Type":"ContainerDied","Data":"8e7fe9e02f3a7ee203f34006c16df36be582063a4852152cedb20e42bfd7f44c"}
Sep 17 21:13:47.463270 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:47.463238    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff44d446f-582ts" event={"ID":"66d48507-65f6-4324-ae1b-0f00cd994c79","Type":"ContainerStarted","Data":"9592677eabfbb60d90dcaad729e0be2ca89bb08fbab1b4e22f2edf4d9ad66457"}
Sep 17 21:13:47.487730 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:47.487680    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff44d446f-582ts" podStartSLOduration=3.004677575 podStartE2EDuration="5.487663573s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.482985998s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:13:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:13:47.486222031 +0000 UTC m=+1492.638666428" watchObservedRunningTime="2026-09-17 21:13:47.487663573 +0000 UTC m=+1492.640107949"
Sep 17 21:13:47.842161 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:47.842066    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff44d446f-582ts"
Sep 17 21:13:49.290166 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:49.290127    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:13:51.857716 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:51.857678    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:13:51.940334 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:51.940280    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:13:52.842356 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:52.842319    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff44d446f-582ts"
Sep 17 21:13:52.843791 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:52.843761    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff44d446f-582ts" podUID="66d48507-65f6-4324-ae1b-0f00cd994c79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Sep 17 21:13:55.053881 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:55.053826    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:13:55.116804 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:55.116773    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:13:55.414330 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:13:55.414219    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:14:02.741245 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:14:02.741200    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:14:02.842377 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:14:02.842332    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff44d446f-582ts" podUID="66d48507-65f6-4324-ae1b-0f00cd994c79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Sep 17 21:14:07.151898 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:14:07.151823    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:14:12.111076 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:14:12.111042    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-x9sck"]
Sep 17 21:14:12.114890 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:14:12.114866    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-x9sck"
Sep 17 21:14:12.117601 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:14:12.117484    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 17 21:14:12.117601 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:14:12.117561    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 17 21:14:12.117601 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:14:12.117561    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 21:14:12.117947 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:14:12.117568    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-qm6fb\""
Sep 17 21:14:12.117947 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:14:12.117650    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Sep 17 21:14:12.118350 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:14:12.118329    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 17 21:14:12.125783 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:14:12.125757    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-x9sck"]
Sep 17 21:14:12.149867 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:14:12.149801    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c873e606-48ae-4cfc-9f01-99a9724bed15-dshm\") pod \"tokenizer-clean-path-test-kserve-569dcdd89-x9sck\" (UID: \"c873e606-48ae-4cfc-9f01-99a9724bed15\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-x9sck"
Sep 17 21:14:12.150045 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:14:12.149873    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c873e606-48ae-4cfc-9f01-99a9724bed15-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-569dcdd89-x9sck\" (UID: \"c873e606-48ae-4cfc-9f01-99a9724bed15\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-x9sck"
Sep 17 21:14:12.150045 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:14:12.149915    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c873e606-48ae-4cfc-9f01-99a9724bed15-home\") pod \"tokenizer-clean-path-test-kserve-569dcdd89-x9sck\" (UID: \"c873e606-48ae-4cfc-9f01-99a9724bed15\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-x9sck"
Sep 17 21:14:12.150045 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:14:12.149967    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c873e606-48ae-4cfc-9f01-99a9724bed15-model-cache\") pod \"tokenizer-clean-path-test-kserve-569dcdd89-x9sck\" (UID: \"c873e606-48ae-4cfc-9f01-99a9724bed15\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-x9sck"
Sep 17 21:14:12.150045 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:14:12.149989    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c873e606-48ae-4cfc-9f01-99a9724bed15-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-569dcdd89-x9sck\" (UID: \"c873e606-48ae-4cfc-9f01-99a9724bed15\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-x9sck"
Sep 17 21:14:12.150045 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:14:12.150005    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c873e606-48ae-4cfc-9f01-99a9724bed15-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-569dcdd89-x9sck\" (UID: \"c873e606-48ae-4cfc-9f01-99a9724bed15\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-x9sck"
Sep 17 21:14:12.150341 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:14:12.150055    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr5rg\" (UniqueName: \"kubernetes.io/projected/c873e606-48ae-4cfc-9f01-99a9724bed15-kube-api-access-rr5rg\") pod \"tokenizer-clean-path-test-kserve-569dcdd89-x9sck\" (UID: \"c873e606-48ae-4cfc-9f01-99a9724bed15\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-x9sck"
Sep 17 21:14:12.150341 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:14:12.150084    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c873e606-48ae-4cfc-9f01-99a9724bed15-tls-certs\") pod \"tokenizer-clean-path-test-kserve-569dcdd89-x9sck\" (UID: \"c873e606-48ae-4cfc-9f01-99a9724bed15\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-x9sck"
Sep 17 21:14:12.200416 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:14:12.200356    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:14:12.251056 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:14:12.251024    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c873e606-48ae-4cfc-9f01-99a9724bed15-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-569dcdd89-x9sck\" (UID: \"c873e606-48ae-4cfc-9f01-99a9724bed15\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-x9sck"
Sep 17 21:14:12.251056 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:14:12.251061    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c873e606-48ae-4cfc-9f01-99a9724bed15-home\") pod \"tokenizer-clean-path-test-kserve-569dcdd89-x9sck\" (UID: \"c873e606-48ae-4cfc-9f01-99a9724bed15\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-x9sck"
Sep 17 21:14:12.251329 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:14:12.251279    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c873e606-48ae-4cfc-9f01-99a9724bed15-model-cache\") pod \"tokenizer-clean-path-test-kserve-569dcdd89-x9sck\" (UID: \"c873e606-48ae-4cfc-9f01-99a9724bed15\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-x9sck"
Sep 17 21:14:12.251401 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:14:12.251339    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c873e606-48ae-4cfc-9f01-99a9724bed15-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-569dcdd89-x9sck\" (UID: \"c873e606-48ae-4cfc-9f01-99a9724bed15\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-x9sck"
Sep 17 21:14:12.251456 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:14:12.251419    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c873e606-48ae-4cfc-9f01-99a9724bed15-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-569dcdd89-x9sck\" (UID: \"c873e606-48ae-4cfc-9f01-99a9724bed15\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-x9sck"
Sep 17 21:14:12.251514 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:14:12.251465    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr5rg\" (UniqueName: \"kubernetes.io/projected/c873e606-48ae-4cfc-9f01-99a9724bed15-kube-api-access-rr5rg\") pod \"tokenizer-clean-path-test-kserve-569dcdd89-x9sck\" (UID: \"c873e606-48ae-4cfc-9f01-99a9724bed15\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-x9sck"
Sep 17 21:14:12.251514 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:14:12.251476    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c873e606-48ae-4cfc-9f01-99a9724bed15-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-569dcdd89-x9sck\" (UID: \"c873e606-48ae-4cfc-9f01-99a9724bed15\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-x9sck"
Sep 17 21:14:12.251514 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:14:12.251493    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c873e606-48ae-4cfc-9f01-99a9724bed15-home\") pod \"tokenizer-clean-path-test-kserve-569dcdd89-x9sck\" (UID: \"c873e606-48ae-4cfc-9f01-99a9724bed15\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-x9sck"
Sep 17 21:14:12.251514 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:14:12.251501    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c873e606-48ae-4cfc-9f01-99a9724bed15-tls-certs\") pod \"tokenizer-clean-path-test-kserve-569dcdd89-x9sck\" (UID: \"c873e606-48ae-4cfc-9f01-99a9724bed15\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-x9sck"
Sep 17 21:14:12.251743 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:14:12.251608    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c873e606-48ae-4cfc-9f01-99a9724bed15-dshm\") pod \"tokenizer-clean-path-test-kserve-569dcdd89-x9sck\" (UID: \"c873e606-48ae-4cfc-9f01-99a9724bed15\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-x9sck"
Sep 17 21:14:12.251743 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:14:12.251655    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c873e606-48ae-4cfc-9f01-99a9724bed15-model-cache\") pod \"tokenizer-clean-path-test-kserve-569dcdd89-x9sck\" (UID: \"c873e606-48ae-4cfc-9f01-99a9724bed15\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-x9sck"
Sep 17 21:14:12.251743 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:14:12.251720    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c873e606-48ae-4cfc-9f01-99a9724bed15-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-569dcdd89-x9sck\" (UID: \"c873e606-48ae-4cfc-9f01-99a9724bed15\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-x9sck"
Sep 17 21:14:12.252272 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:14:12.252245    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c873e606-48ae-4cfc-9f01-99a9724bed15-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-569dcdd89-x9sck\" (UID: \"c873e606-48ae-4cfc-9f01-99a9724bed15\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-x9sck"
Sep 17 21:14:12.253736 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:14:12.253715    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c873e606-48ae-4cfc-9f01-99a9724bed15-dshm\") pod \"tokenizer-clean-path-test-kserve-569dcdd89-x9sck\" (UID: \"c873e606-48ae-4cfc-9f01-99a9724bed15\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-x9sck"
Sep 17 21:14:12.254018 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:14:12.254000    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c873e606-48ae-4cfc-9f01-99a9724bed15-tls-certs\") pod \"tokenizer-clean-path-test-kserve-569dcdd89-x9sck\" (UID: \"c873e606-48ae-4cfc-9f01-99a9724bed15\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-x9sck"
Sep 17 21:14:12.259521 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:14:12.259500    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr5rg\" (UniqueName: \"kubernetes.io/projected/c873e606-48ae-4cfc-9f01-99a9724bed15-kube-api-access-rr5rg\") pod \"tokenizer-clean-path-test-kserve-569dcdd89-x9sck\" (UID: \"c873e606-48ae-4cfc-9f01-99a9724bed15\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-x9sck"
Sep 17 21:14:12.407553 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:14:12.407447    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:14:12.426340 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:14:12.426303    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-x9sck"
Sep 17 21:14:12.471090 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:14:12.471033    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:14:12.709999 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:14:12.709954    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-x9sck"]
Sep 17 21:14:12.713779 ip-10-0-139-36 kubenswrapper[2575]: W0917 21:14:12.713740    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc873e606_48ae_4cfc_9f01_99a9724bed15.slice/crio-0e51df53a74a6ccbeeaed1f08aac111b7ff0d35362c7201cd9cfded38a5eeac6 WatchSource:0}: Error finding container 0e51df53a74a6ccbeeaed1f08aac111b7ff0d35362c7201cd9cfded38a5eeac6: Status 404 returned error can't find the container with id 0e51df53a74a6ccbeeaed1f08aac111b7ff0d35362c7201cd9cfded38a5eeac6
Sep 17 21:14:12.842986 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:14:12.842936    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff44d446f-582ts" podUID="66d48507-65f6-4324-ae1b-0f00cd994c79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Sep 17 21:14:13.606958 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:14:13.606916    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-x9sck" event={"ID":"c873e606-48ae-4cfc-9f01-99a9724bed15","Type":"ContainerStarted","Data":"819a4f2363e8cb2eff6a243c73fa0b9cc0b13e1469de5f007f97a07c647f4200"}
Sep 17 21:14:13.607369 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:14:13.606991    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-x9sck" event={"ID":"c873e606-48ae-4cfc-9f01-99a9724bed15","Type":"ContainerStarted","Data":"0e51df53a74a6ccbeeaed1f08aac111b7ff0d35362c7201cd9cfded38a5eeac6"}
Sep 17 21:14:15.620452 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:14:15.620364    2575 generic.go:356] "Generic (PLEG): container finished" podID="c873e606-48ae-4cfc-9f01-99a9724bed15" containerID="819a4f2363e8cb2eff6a243c73fa0b9cc0b13e1469de5f007f97a07c647f4200" exitCode=0
Sep 17 21:14:15.620452 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:14:15.620438    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-x9sck" event={"ID":"c873e606-48ae-4cfc-9f01-99a9724bed15","Type":"ContainerDied","Data":"819a4f2363e8cb2eff6a243c73fa0b9cc0b13e1469de5f007f97a07c647f4200"}
Sep 17 21:14:16.628596 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:14:16.628563    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-x9sck" event={"ID":"c873e606-48ae-4cfc-9f01-99a9724bed15","Type":"ContainerStarted","Data":"598bb1917c9b2cea826deb927a601876bf27d5c93ce2de9d76b1b96365f50e14"}
Sep 17 21:14:16.649446 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:14:16.649390    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-x9sck" podStartSLOduration=2.196709487 podStartE2EDuration="4.649373246s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.452663759s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:14:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:14:16.647319284 +0000 UTC m=+1521.799763659" watchObservedRunningTime="2026-09-17 21:14:16.649373246 +0000 UTC m=+1521.801817622"
Sep 17 21:14:17.427444 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:14:17.427400    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-x9sck"
Sep 17 21:14:22.427140 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:14:22.427044    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-x9sck"
Sep 17 21:14:22.440024 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:14:22.439996    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-x9sck"
Sep 17 21:14:22.673103 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:14:22.673069    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-x9sck"
Sep 17 21:14:22.842976 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:14:22.842933    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff44d446f-582ts" podUID="66d48507-65f6-4324-ae1b-0f00cd994c79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Sep 17 21:14:31.427481 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:14:31.427438    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:14:32.842645 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:14:32.842606    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff44d446f-582ts" podUID="66d48507-65f6-4324-ae1b-0f00cd994c79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Sep 17 21:14:42.842043 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:14:42.841996    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff44d446f-582ts" podUID="66d48507-65f6-4324-ae1b-0f00cd994c79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Sep 17 21:14:52.843045 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:14:52.842999    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff44d446f-582ts" podUID="66d48507-65f6-4324-ae1b-0f00cd994c79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Sep 17 21:15:02.842913 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:15:02.842837    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff44d446f-582ts" podUID="66d48507-65f6-4324-ae1b-0f00cd994c79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Sep 17 21:15:12.842888 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:15:12.842821    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff44d446f-582ts" podUID="66d48507-65f6-4324-ae1b-0f00cd994c79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Sep 17 21:15:22.842068 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:15:22.842019    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff44d446f-582ts" podUID="66d48507-65f6-4324-ae1b-0f00cd994c79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Sep 17 21:15:32.842142 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:15:32.842094    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff44d446f-582ts" podUID="66d48507-65f6-4324-ae1b-0f00cd994c79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Sep 17 21:15:40.154627 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:15:40.154585    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-x9sck"]
Sep 17 21:15:40.164450 ip-10-0-139-36 kubenswrapper[2575]: E0917 21:15:40.164407    2575 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 21:15:40.164450 ip-10-0-139-36 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:15:40.164450 ip-10-0-139-36 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 17 21:15:40.164450 ip-10-0-139-36 kubenswrapper[2575]:  > containerID="598bb1917c9b2cea826deb927a601876bf27d5c93ce2de9d76b1b96365f50e14" cmd=["/bin/sleep","15"]
Sep 17 21:15:40.164787 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:15:40.164463    2575 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-x9sck" message="" err=<
Sep 17 21:15:40.164787 ip-10-0-139-36 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:15:40.164787 ip-10-0-139-36 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 17 21:15:40.164787 ip-10-0-139-36 kubenswrapper[2575]:  >
Sep 17 21:15:40.164787 ip-10-0-139-36 kubenswrapper[2575]: E0917 21:15:40.164476    2575 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 21:15:40.164787 ip-10-0-139-36 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:15:40.164787 ip-10-0-139-36 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 17 21:15:40.164787 ip-10-0-139-36 kubenswrapper[2575]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-x9sck" podUID="c873e606-48ae-4cfc-9f01-99a9724bed15" containerName="main" containerID="cri-o://598bb1917c9b2cea826deb927a601876bf27d5c93ce2de9d76b1b96365f50e14"
Sep 17 21:15:40.164787 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:15:40.164540    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-x9sck" podUID="c873e606-48ae-4cfc-9f01-99a9724bed15" containerName="main" containerID="cri-o://598bb1917c9b2cea826deb927a601876bf27d5c93ce2de9d76b1b96365f50e14" gracePeriod=60
Sep 17 21:15:40.520315 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:15:40.520287    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-x9sck"
Sep 17 21:15:40.612861 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:15:40.612809    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c873e606-48ae-4cfc-9f01-99a9724bed15-tls-certs\") pod \"c873e606-48ae-4cfc-9f01-99a9724bed15\" (UID: \"c873e606-48ae-4cfc-9f01-99a9724bed15\") "
Sep 17 21:15:40.613044 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:15:40.612916    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c873e606-48ae-4cfc-9f01-99a9724bed15-cabundle-cert\") pod \"c873e606-48ae-4cfc-9f01-99a9724bed15\" (UID: \"c873e606-48ae-4cfc-9f01-99a9724bed15\") "
Sep 17 21:15:40.613044 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:15:40.612944    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c873e606-48ae-4cfc-9f01-99a9724bed15-tmp-dir\") pod \"c873e606-48ae-4cfc-9f01-99a9724bed15\" (UID: \"c873e606-48ae-4cfc-9f01-99a9724bed15\") "
Sep 17 21:15:40.613044 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:15:40.612970    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c873e606-48ae-4cfc-9f01-99a9724bed15-kserve-provision-location\") pod \"c873e606-48ae-4cfc-9f01-99a9724bed15\" (UID: \"c873e606-48ae-4cfc-9f01-99a9724bed15\") "
Sep 17 21:15:40.613216 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:15:40.613046    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c873e606-48ae-4cfc-9f01-99a9724bed15-dshm\") pod \"c873e606-48ae-4cfc-9f01-99a9724bed15\" (UID: \"c873e606-48ae-4cfc-9f01-99a9724bed15\") "
Sep 17 21:15:40.613216 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:15:40.613078    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rr5rg\" (UniqueName: \"kubernetes.io/projected/c873e606-48ae-4cfc-9f01-99a9724bed15-kube-api-access-rr5rg\") pod \"c873e606-48ae-4cfc-9f01-99a9724bed15\" (UID: \"c873e606-48ae-4cfc-9f01-99a9724bed15\") "
Sep 17 21:15:40.613216 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:15:40.613153    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c873e606-48ae-4cfc-9f01-99a9724bed15-home\") pod \"c873e606-48ae-4cfc-9f01-99a9724bed15\" (UID: \"c873e606-48ae-4cfc-9f01-99a9724bed15\") "
Sep 17 21:15:40.613216 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:15:40.613178    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c873e606-48ae-4cfc-9f01-99a9724bed15-model-cache\") pod \"c873e606-48ae-4cfc-9f01-99a9724bed15\" (UID: \"c873e606-48ae-4cfc-9f01-99a9724bed15\") "
Sep 17 21:15:40.613216 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:15:40.613190    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c873e606-48ae-4cfc-9f01-99a9724bed15-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c873e606-48ae-4cfc-9f01-99a9724bed15" (UID: "c873e606-48ae-4cfc-9f01-99a9724bed15"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:15:40.613533 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:15:40.613334    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c873e606-48ae-4cfc-9f01-99a9724bed15-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c873e606-48ae-4cfc-9f01-99a9724bed15" (UID: "c873e606-48ae-4cfc-9f01-99a9724bed15"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:15:40.613533 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:15:40.613379    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c873e606-48ae-4cfc-9f01-99a9724bed15-home" (OuterVolumeSpecName: "home") pod "c873e606-48ae-4cfc-9f01-99a9724bed15" (UID: "c873e606-48ae-4cfc-9f01-99a9724bed15"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:15:40.613533 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:15:40.613492    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c873e606-48ae-4cfc-9f01-99a9724bed15-cabundle-cert\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:15:40.613533 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:15:40.613510    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c873e606-48ae-4cfc-9f01-99a9724bed15-tmp-dir\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:15:40.613748 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:15:40.613604    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c873e606-48ae-4cfc-9f01-99a9724bed15-model-cache" (OuterVolumeSpecName: "model-cache") pod "c873e606-48ae-4cfc-9f01-99a9724bed15" (UID: "c873e606-48ae-4cfc-9f01-99a9724bed15"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:15:40.615465 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:15:40.615438    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c873e606-48ae-4cfc-9f01-99a9724bed15-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c873e606-48ae-4cfc-9f01-99a9724bed15" (UID: "c873e606-48ae-4cfc-9f01-99a9724bed15"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:15:40.615465 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:15:40.615454    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c873e606-48ae-4cfc-9f01-99a9724bed15-kube-api-access-rr5rg" (OuterVolumeSpecName: "kube-api-access-rr5rg") pod "c873e606-48ae-4cfc-9f01-99a9724bed15" (UID: "c873e606-48ae-4cfc-9f01-99a9724bed15"). InnerVolumeSpecName "kube-api-access-rr5rg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:15:40.615637 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:15:40.615464    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c873e606-48ae-4cfc-9f01-99a9724bed15-dshm" (OuterVolumeSpecName: "dshm") pod "c873e606-48ae-4cfc-9f01-99a9724bed15" (UID: "c873e606-48ae-4cfc-9f01-99a9724bed15"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:15:40.632497 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:15:40.632467    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c873e606-48ae-4cfc-9f01-99a9724bed15-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c873e606-48ae-4cfc-9f01-99a9724bed15" (UID: "c873e606-48ae-4cfc-9f01-99a9724bed15"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:15:40.714796 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:15:40.714760    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c873e606-48ae-4cfc-9f01-99a9724bed15-kserve-provision-location\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:15:40.714796 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:15:40.714790    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c873e606-48ae-4cfc-9f01-99a9724bed15-dshm\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:15:40.714796 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:15:40.714802    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rr5rg\" (UniqueName: \"kubernetes.io/projected/c873e606-48ae-4cfc-9f01-99a9724bed15-kube-api-access-rr5rg\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:15:40.715083 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:15:40.714810    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c873e606-48ae-4cfc-9f01-99a9724bed15-home\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:15:40.715083 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:15:40.714819    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c873e606-48ae-4cfc-9f01-99a9724bed15-model-cache\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:15:40.715083 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:15:40.714826    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c873e606-48ae-4cfc-9f01-99a9724bed15-tls-certs\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:15:40.728235 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:15:40.728204    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:15:40.788504 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:15:40.788467    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:15:41.097409 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:15:41.097312    2575 generic.go:356] "Generic (PLEG): container finished" podID="c873e606-48ae-4cfc-9f01-99a9724bed15" containerID="598bb1917c9b2cea826deb927a601876bf27d5c93ce2de9d76b1b96365f50e14" exitCode=0
Sep 17 21:15:41.097409 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:15:41.097364    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-x9sck" event={"ID":"c873e606-48ae-4cfc-9f01-99a9724bed15","Type":"ContainerDied","Data":"598bb1917c9b2cea826deb927a601876bf27d5c93ce2de9d76b1b96365f50e14"}
Sep 17 21:15:41.097627 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:15:41.097377    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-x9sck"
Sep 17 21:15:41.097627 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:15:41.097476    2575 scope.go:117] "RemoveContainer" containerID="598bb1917c9b2cea826deb927a601876bf27d5c93ce2de9d76b1b96365f50e14"
Sep 17 21:15:41.097627 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:15:41.097463    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-x9sck" event={"ID":"c873e606-48ae-4cfc-9f01-99a9724bed15","Type":"ContainerDied","Data":"0e51df53a74a6ccbeeaed1f08aac111b7ff0d35362c7201cd9cfded38a5eeac6"}
Sep 17 21:15:41.107247 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:15:41.107229    2575 scope.go:117] "RemoveContainer" containerID="819a4f2363e8cb2eff6a243c73fa0b9cc0b13e1469de5f007f97a07c647f4200"
Sep 17 21:15:41.115217 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:15:41.115193    2575 scope.go:117] "RemoveContainer" containerID="598bb1917c9b2cea826deb927a601876bf27d5c93ce2de9d76b1b96365f50e14"
Sep 17 21:15:41.115493 ip-10-0-139-36 kubenswrapper[2575]: E0917 21:15:41.115471    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"598bb1917c9b2cea826deb927a601876bf27d5c93ce2de9d76b1b96365f50e14\": container with ID starting with 598bb1917c9b2cea826deb927a601876bf27d5c93ce2de9d76b1b96365f50e14 not found: ID does not exist" containerID="598bb1917c9b2cea826deb927a601876bf27d5c93ce2de9d76b1b96365f50e14"
Sep 17 21:15:41.115578 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:15:41.115506    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"598bb1917c9b2cea826deb927a601876bf27d5c93ce2de9d76b1b96365f50e14"} err="failed to get container status \"598bb1917c9b2cea826deb927a601876bf27d5c93ce2de9d76b1b96365f50e14\": rpc error: code = NotFound desc = could not find container \"598bb1917c9b2cea826deb927a601876bf27d5c93ce2de9d76b1b96365f50e14\": container with ID starting with 598bb1917c9b2cea826deb927a601876bf27d5c93ce2de9d76b1b96365f50e14 not found: ID does not exist"
Sep 17 21:15:41.115578 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:15:41.115531    2575 scope.go:117] "RemoveContainer" containerID="819a4f2363e8cb2eff6a243c73fa0b9cc0b13e1469de5f007f97a07c647f4200"
Sep 17 21:15:41.115784 ip-10-0-139-36 kubenswrapper[2575]: E0917 21:15:41.115764    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"819a4f2363e8cb2eff6a243c73fa0b9cc0b13e1469de5f007f97a07c647f4200\": container with ID starting with 819a4f2363e8cb2eff6a243c73fa0b9cc0b13e1469de5f007f97a07c647f4200 not found: ID does not exist" containerID="819a4f2363e8cb2eff6a243c73fa0b9cc0b13e1469de5f007f97a07c647f4200"
Sep 17 21:15:41.115836 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:15:41.115793    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"819a4f2363e8cb2eff6a243c73fa0b9cc0b13e1469de5f007f97a07c647f4200"} err="failed to get container status \"819a4f2363e8cb2eff6a243c73fa0b9cc0b13e1469de5f007f97a07c647f4200\": rpc error: code = NotFound desc = could not find container \"819a4f2363e8cb2eff6a243c73fa0b9cc0b13e1469de5f007f97a07c647f4200\": container with ID starting with 819a4f2363e8cb2eff6a243c73fa0b9cc0b13e1469de5f007f97a07c647f4200 not found: ID does not exist"
Sep 17 21:15:41.118323 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:15:41.118302    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-x9sck"]
Sep 17 21:15:41.122454 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:15:41.122430    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-569dcdd89-x9sck"]
Sep 17 21:15:41.356976 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:15:41.356894    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c873e606-48ae-4cfc-9f01-99a9724bed15" path="/var/lib/kubelet/pods/c873e606-48ae-4cfc-9f01-99a9724bed15/volumes"
Sep 17 21:15:42.852635 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:15:42.852605    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff44d446f-582ts"
Sep 17 21:15:42.860448 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:15:42.860422    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff44d446f-582ts"
Sep 17 21:15:44.559750 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:15:44.559709    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:15:44.617051 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:15:44.617020    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff44d446f-582ts"]
Sep 17 21:15:44.656180 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:15:44.656144    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:15:45.895232 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:15:45.895186    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:15:59.639759 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:15:59.639717    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff44d446f-582ts" podUID="66d48507-65f6-4324-ae1b-0f00cd994c79" containerName="main" containerID="cri-o://9592677eabfbb60d90dcaad729e0be2ca89bb08fbab1b4e22f2edf4d9ad66457" gracePeriod=45
Sep 17 21:15:59.854171 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:15:59.854127    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff44d446f-582ts" podUID="66d48507-65f6-4324-ae1b-0f00cd994c79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Sep 17 21:16:00.395350 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:00.395326    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff44d446f-582ts"
Sep 17 21:16:00.493199 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:00.493169    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66d48507-65f6-4324-ae1b-0f00cd994c79-home\") pod \"66d48507-65f6-4324-ae1b-0f00cd994c79\" (UID: \"66d48507-65f6-4324-ae1b-0f00cd994c79\") "
Sep 17 21:16:00.493199 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:00.493199    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66d48507-65f6-4324-ae1b-0f00cd994c79-dshm\") pod \"66d48507-65f6-4324-ae1b-0f00cd994c79\" (UID: \"66d48507-65f6-4324-ae1b-0f00cd994c79\") "
Sep 17 21:16:00.493449 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:00.493232    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66d48507-65f6-4324-ae1b-0f00cd994c79-cabundle-cert\") pod \"66d48507-65f6-4324-ae1b-0f00cd994c79\" (UID: \"66d48507-65f6-4324-ae1b-0f00cd994c79\") "
Sep 17 21:16:00.493449 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:00.493275    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66d48507-65f6-4324-ae1b-0f00cd994c79-tmp-dir\") pod \"66d48507-65f6-4324-ae1b-0f00cd994c79\" (UID: \"66d48507-65f6-4324-ae1b-0f00cd994c79\") "
Sep 17 21:16:00.493449 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:00.493321    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66d48507-65f6-4324-ae1b-0f00cd994c79-kserve-provision-location\") pod \"66d48507-65f6-4324-ae1b-0f00cd994c79\" (UID: \"66d48507-65f6-4324-ae1b-0f00cd994c79\") "
Sep 17 21:16:00.493449 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:00.493352    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66d48507-65f6-4324-ae1b-0f00cd994c79-tls-certs\") pod \"66d48507-65f6-4324-ae1b-0f00cd994c79\" (UID: \"66d48507-65f6-4324-ae1b-0f00cd994c79\") "
Sep 17 21:16:00.493449 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:00.493395    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xg54p\" (UniqueName: \"kubernetes.io/projected/66d48507-65f6-4324-ae1b-0f00cd994c79-kube-api-access-xg54p\") pod \"66d48507-65f6-4324-ae1b-0f00cd994c79\" (UID: \"66d48507-65f6-4324-ae1b-0f00cd994c79\") "
Sep 17 21:16:00.493714 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:00.493464    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66d48507-65f6-4324-ae1b-0f00cd994c79-model-cache\") pod \"66d48507-65f6-4324-ae1b-0f00cd994c79\" (UID: \"66d48507-65f6-4324-ae1b-0f00cd994c79\") "
Sep 17 21:16:00.493714 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:00.493675    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66d48507-65f6-4324-ae1b-0f00cd994c79-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "66d48507-65f6-4324-ae1b-0f00cd994c79" (UID: "66d48507-65f6-4324-ae1b-0f00cd994c79"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:16:00.493931 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:00.493790    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66d48507-65f6-4324-ae1b-0f00cd994c79-cabundle-cert\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:16:00.494154 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:00.494121    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66d48507-65f6-4324-ae1b-0f00cd994c79-model-cache" (OuterVolumeSpecName: "model-cache") pod "66d48507-65f6-4324-ae1b-0f00cd994c79" (UID: "66d48507-65f6-4324-ae1b-0f00cd994c79"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:16:00.494280 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:00.494247    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66d48507-65f6-4324-ae1b-0f00cd994c79-home" (OuterVolumeSpecName: "home") pod "66d48507-65f6-4324-ae1b-0f00cd994c79" (UID: "66d48507-65f6-4324-ae1b-0f00cd994c79"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:16:00.496090 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:00.496058    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66d48507-65f6-4324-ae1b-0f00cd994c79-dshm" (OuterVolumeSpecName: "dshm") pod "66d48507-65f6-4324-ae1b-0f00cd994c79" (UID: "66d48507-65f6-4324-ae1b-0f00cd994c79"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:16:00.496090 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:00.496063    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66d48507-65f6-4324-ae1b-0f00cd994c79-kube-api-access-xg54p" (OuterVolumeSpecName: "kube-api-access-xg54p") pod "66d48507-65f6-4324-ae1b-0f00cd994c79" (UID: "66d48507-65f6-4324-ae1b-0f00cd994c79"). InnerVolumeSpecName "kube-api-access-xg54p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:16:00.496226 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:00.496083    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66d48507-65f6-4324-ae1b-0f00cd994c79-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "66d48507-65f6-4324-ae1b-0f00cd994c79" (UID: "66d48507-65f6-4324-ae1b-0f00cd994c79"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:16:00.505724 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:00.505699    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66d48507-65f6-4324-ae1b-0f00cd994c79-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "66d48507-65f6-4324-ae1b-0f00cd994c79" (UID: "66d48507-65f6-4324-ae1b-0f00cd994c79"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:16:00.517199 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:00.517176    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66d48507-65f6-4324-ae1b-0f00cd994c79-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "66d48507-65f6-4324-ae1b-0f00cd994c79" (UID: "66d48507-65f6-4324-ae1b-0f00cd994c79"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:16:00.594376 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:00.594343    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66d48507-65f6-4324-ae1b-0f00cd994c79-home\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:16:00.594376 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:00.594370    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66d48507-65f6-4324-ae1b-0f00cd994c79-dshm\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:16:00.594376 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:00.594379    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66d48507-65f6-4324-ae1b-0f00cd994c79-tmp-dir\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:16:00.594594 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:00.594390    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66d48507-65f6-4324-ae1b-0f00cd994c79-kserve-provision-location\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:16:00.594594 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:00.594400    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66d48507-65f6-4324-ae1b-0f00cd994c79-tls-certs\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:16:00.594594 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:00.594409    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xg54p\" (UniqueName: \"kubernetes.io/projected/66d48507-65f6-4324-ae1b-0f00cd994c79-kube-api-access-xg54p\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:16:00.594594 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:00.594417    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66d48507-65f6-4324-ae1b-0f00cd994c79-model-cache\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:16:01.205392 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:01.205360    2575 generic.go:356] "Generic (PLEG): container finished" podID="66d48507-65f6-4324-ae1b-0f00cd994c79" containerID="9592677eabfbb60d90dcaad729e0be2ca89bb08fbab1b4e22f2edf4d9ad66457" exitCode=0
Sep 17 21:16:01.205822 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:01.205415    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff44d446f-582ts" event={"ID":"66d48507-65f6-4324-ae1b-0f00cd994c79","Type":"ContainerDied","Data":"9592677eabfbb60d90dcaad729e0be2ca89bb08fbab1b4e22f2edf4d9ad66457"}
Sep 17 21:16:01.205822 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:01.205421    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff44d446f-582ts"
Sep 17 21:16:01.205822 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:01.205456    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff44d446f-582ts" event={"ID":"66d48507-65f6-4324-ae1b-0f00cd994c79","Type":"ContainerDied","Data":"5e81f99c0b9a9543ed15b98c7d983319cc82845ee139ca3e44288f8ecfa4abaf"}
Sep 17 21:16:01.205822 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:01.205477    2575 scope.go:117] "RemoveContainer" containerID="9592677eabfbb60d90dcaad729e0be2ca89bb08fbab1b4e22f2edf4d9ad66457"
Sep 17 21:16:01.214872 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:01.214835    2575 scope.go:117] "RemoveContainer" containerID="8e7fe9e02f3a7ee203f34006c16df36be582063a4852152cedb20e42bfd7f44c"
Sep 17 21:16:01.222919 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:01.222900    2575 scope.go:117] "RemoveContainer" containerID="9592677eabfbb60d90dcaad729e0be2ca89bb08fbab1b4e22f2edf4d9ad66457"
Sep 17 21:16:01.223213 ip-10-0-139-36 kubenswrapper[2575]: E0917 21:16:01.223185    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9592677eabfbb60d90dcaad729e0be2ca89bb08fbab1b4e22f2edf4d9ad66457\": container with ID starting with 9592677eabfbb60d90dcaad729e0be2ca89bb08fbab1b4e22f2edf4d9ad66457 not found: ID does not exist" containerID="9592677eabfbb60d90dcaad729e0be2ca89bb08fbab1b4e22f2edf4d9ad66457"
Sep 17 21:16:01.223261 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:01.223221    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9592677eabfbb60d90dcaad729e0be2ca89bb08fbab1b4e22f2edf4d9ad66457"} err="failed to get container status \"9592677eabfbb60d90dcaad729e0be2ca89bb08fbab1b4e22f2edf4d9ad66457\": rpc error: code = NotFound desc = could not find container \"9592677eabfbb60d90dcaad729e0be2ca89bb08fbab1b4e22f2edf4d9ad66457\": container with ID starting with 9592677eabfbb60d90dcaad729e0be2ca89bb08fbab1b4e22f2edf4d9ad66457 not found: ID does not exist"
Sep 17 21:16:01.223261 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:01.223237    2575 scope.go:117] "RemoveContainer" containerID="8e7fe9e02f3a7ee203f34006c16df36be582063a4852152cedb20e42bfd7f44c"
Sep 17 21:16:01.223532 ip-10-0-139-36 kubenswrapper[2575]: E0917 21:16:01.223511    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e7fe9e02f3a7ee203f34006c16df36be582063a4852152cedb20e42bfd7f44c\": container with ID starting with 8e7fe9e02f3a7ee203f34006c16df36be582063a4852152cedb20e42bfd7f44c not found: ID does not exist" containerID="8e7fe9e02f3a7ee203f34006c16df36be582063a4852152cedb20e42bfd7f44c"
Sep 17 21:16:01.223587 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:01.223537    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e7fe9e02f3a7ee203f34006c16df36be582063a4852152cedb20e42bfd7f44c"} err="failed to get container status \"8e7fe9e02f3a7ee203f34006c16df36be582063a4852152cedb20e42bfd7f44c\": rpc error: code = NotFound desc = could not find container \"8e7fe9e02f3a7ee203f34006c16df36be582063a4852152cedb20e42bfd7f44c\": container with ID starting with 8e7fe9e02f3a7ee203f34006c16df36be582063a4852152cedb20e42bfd7f44c not found: ID does not exist"
Sep 17 21:16:01.228329 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:01.228291    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff44d446f-582ts"]
Sep 17 21:16:01.231678 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:01.231656    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff44d446f-582ts"]
Sep 17 21:16:01.356912 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:01.356880    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66d48507-65f6-4324-ae1b-0f00cd994c79" path="/var/lib/kubelet/pods/66d48507-65f6-4324-ae1b-0f00cd994c79/volumes"
Sep 17 21:16:30.209612 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:30.209570    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:16:30.300605 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:30.300565    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:16:34.274004 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:34.273968    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:16:34.378134 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:34.378083    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:16:37.127078 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:37.127027    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:16:37.457166 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:37.457135    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:16:37.555081 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:37.555040    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:16:38.131739 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:38.131698    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:16:41.765694 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:41.765658    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:16:43.960227 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:43.960191    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:16:44.086607 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:44.086575    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:16:46.144514 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:46.144483    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:16:46.654163 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:46.654126    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:16:48.252425 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:48.252379    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:16:49.381260 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:49.381221    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:16:55.754131 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:55.754048    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c4d7f586-tj7fp"]
Sep 17 21:16:55.754597 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:55.754576    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c873e606-48ae-4cfc-9f01-99a9724bed15" containerName="main"
Sep 17 21:16:55.754676 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:55.754600    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c873e606-48ae-4cfc-9f01-99a9724bed15" containerName="main"
Sep 17 21:16:55.754676 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:55.754615    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66d48507-65f6-4324-ae1b-0f00cd994c79" containerName="main"
Sep 17 21:16:55.754676 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:55.754623    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="66d48507-65f6-4324-ae1b-0f00cd994c79" containerName="main"
Sep 17 21:16:55.754676 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:55.754654    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66d48507-65f6-4324-ae1b-0f00cd994c79" containerName="storage-initializer"
Sep 17 21:16:55.754676 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:55.754664    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="66d48507-65f6-4324-ae1b-0f00cd994c79" containerName="storage-initializer"
Sep 17 21:16:55.754676 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:55.754673    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c873e606-48ae-4cfc-9f01-99a9724bed15" containerName="storage-initializer"
Sep 17 21:16:55.755013 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:55.754683    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c873e606-48ae-4cfc-9f01-99a9724bed15" containerName="storage-initializer"
Sep 17 21:16:55.755013 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:55.754791    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="66d48507-65f6-4324-ae1b-0f00cd994c79" containerName="main"
Sep 17 21:16:55.755013 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:55.754803    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c873e606-48ae-4cfc-9f01-99a9724bed15" containerName="main"
Sep 17 21:16:55.759321 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:55.759298    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c4d7f586-tj7fp"
Sep 17 21:16:55.761978 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:55.761956    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 21:16:55.762371 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:55.762348    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Sep 17 21:16:55.762487 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:55.762414    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Sep 17 21:16:55.762559 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:55.762537    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"seaweedfs-s3-creds\""
Sep 17 21:16:55.762711 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:55.762690    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-2znt5\""
Sep 17 21:16:55.762812 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:55.762774    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Sep 17 21:16:55.776269 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:55.776245    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c4d7f586-tj7fp"]
Sep 17 21:16:55.777979 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:55.777948    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7a93e7f-28de-4f72-9ae2-15f8ca15be89-home\") pod \"auth-disabled-test-kserve-77c4d7f586-tj7fp\" (UID: \"b7a93e7f-28de-4f72-9ae2-15f8ca15be89\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c4d7f586-tj7fp"
Sep 17 21:16:55.778082 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:55.777982    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7a93e7f-28de-4f72-9ae2-15f8ca15be89-dshm\") pod \"auth-disabled-test-kserve-77c4d7f586-tj7fp\" (UID: \"b7a93e7f-28de-4f72-9ae2-15f8ca15be89\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c4d7f586-tj7fp"
Sep 17 21:16:55.778082 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:55.778027    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7a93e7f-28de-4f72-9ae2-15f8ca15be89-model-cache\") pod \"auth-disabled-test-kserve-77c4d7f586-tj7fp\" (UID: \"b7a93e7f-28de-4f72-9ae2-15f8ca15be89\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c4d7f586-tj7fp"
Sep 17 21:16:55.778168 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:55.778113    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7a93e7f-28de-4f72-9ae2-15f8ca15be89-tls-certs\") pod \"auth-disabled-test-kserve-77c4d7f586-tj7fp\" (UID: \"b7a93e7f-28de-4f72-9ae2-15f8ca15be89\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c4d7f586-tj7fp"
Sep 17 21:16:55.778168 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:55.778156    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b7a93e7f-28de-4f72-9ae2-15f8ca15be89-cabundle-cert\") pod \"auth-disabled-test-kserve-77c4d7f586-tj7fp\" (UID: \"b7a93e7f-28de-4f72-9ae2-15f8ca15be89\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c4d7f586-tj7fp"
Sep 17 21:16:55.778233 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:55.778183    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7a93e7f-28de-4f72-9ae2-15f8ca15be89-tmp-dir\") pod \"auth-disabled-test-kserve-77c4d7f586-tj7fp\" (UID: \"b7a93e7f-28de-4f72-9ae2-15f8ca15be89\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c4d7f586-tj7fp"
Sep 17 21:16:55.778268 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:55.778258    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq4nt\" (UniqueName: \"kubernetes.io/projected/b7a93e7f-28de-4f72-9ae2-15f8ca15be89-kube-api-access-lq4nt\") pod \"auth-disabled-test-kserve-77c4d7f586-tj7fp\" (UID: \"b7a93e7f-28de-4f72-9ae2-15f8ca15be89\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c4d7f586-tj7fp"
Sep 17 21:16:55.778492 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:55.778302    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b7a93e7f-28de-4f72-9ae2-15f8ca15be89-kserve-provision-location\") pod \"auth-disabled-test-kserve-77c4d7f586-tj7fp\" (UID: \"b7a93e7f-28de-4f72-9ae2-15f8ca15be89\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c4d7f586-tj7fp"
Sep 17 21:16:55.809078 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:55.809054    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:16:55.879339 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:55.879313    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq4nt\" (UniqueName: \"kubernetes.io/projected/b7a93e7f-28de-4f72-9ae2-15f8ca15be89-kube-api-access-lq4nt\") pod \"auth-disabled-test-kserve-77c4d7f586-tj7fp\" (UID: \"b7a93e7f-28de-4f72-9ae2-15f8ca15be89\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c4d7f586-tj7fp"
Sep 17 21:16:55.879449 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:55.879349    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b7a93e7f-28de-4f72-9ae2-15f8ca15be89-kserve-provision-location\") pod \"auth-disabled-test-kserve-77c4d7f586-tj7fp\" (UID: \"b7a93e7f-28de-4f72-9ae2-15f8ca15be89\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c4d7f586-tj7fp"
Sep 17 21:16:55.879449 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:55.879372    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7a93e7f-28de-4f72-9ae2-15f8ca15be89-home\") pod \"auth-disabled-test-kserve-77c4d7f586-tj7fp\" (UID: \"b7a93e7f-28de-4f72-9ae2-15f8ca15be89\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c4d7f586-tj7fp"
Sep 17 21:16:55.879572 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:55.879490    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7a93e7f-28de-4f72-9ae2-15f8ca15be89-dshm\") pod \"auth-disabled-test-kserve-77c4d7f586-tj7fp\" (UID: \"b7a93e7f-28de-4f72-9ae2-15f8ca15be89\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c4d7f586-tj7fp"
Sep 17 21:16:55.879572 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:55.879561    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7a93e7f-28de-4f72-9ae2-15f8ca15be89-model-cache\") pod \"auth-disabled-test-kserve-77c4d7f586-tj7fp\" (UID: \"b7a93e7f-28de-4f72-9ae2-15f8ca15be89\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c4d7f586-tj7fp"
Sep 17 21:16:55.879678 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:55.879608    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7a93e7f-28de-4f72-9ae2-15f8ca15be89-tls-certs\") pod \"auth-disabled-test-kserve-77c4d7f586-tj7fp\" (UID: \"b7a93e7f-28de-4f72-9ae2-15f8ca15be89\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c4d7f586-tj7fp"
Sep 17 21:16:55.879678 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:55.879648    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b7a93e7f-28de-4f72-9ae2-15f8ca15be89-cabundle-cert\") pod \"auth-disabled-test-kserve-77c4d7f586-tj7fp\" (UID: \"b7a93e7f-28de-4f72-9ae2-15f8ca15be89\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c4d7f586-tj7fp"
Sep 17 21:16:55.879769 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:55.879677    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7a93e7f-28de-4f72-9ae2-15f8ca15be89-tmp-dir\") pod \"auth-disabled-test-kserve-77c4d7f586-tj7fp\" (UID: \"b7a93e7f-28de-4f72-9ae2-15f8ca15be89\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c4d7f586-tj7fp"
Sep 17 21:16:55.879769 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:55.879709    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b7a93e7f-28de-4f72-9ae2-15f8ca15be89-kserve-provision-location\") pod \"auth-disabled-test-kserve-77c4d7f586-tj7fp\" (UID: \"b7a93e7f-28de-4f72-9ae2-15f8ca15be89\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c4d7f586-tj7fp"
Sep 17 21:16:55.879769 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:55.879751    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7a93e7f-28de-4f72-9ae2-15f8ca15be89-home\") pod \"auth-disabled-test-kserve-77c4d7f586-tj7fp\" (UID: \"b7a93e7f-28de-4f72-9ae2-15f8ca15be89\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c4d7f586-tj7fp"
Sep 17 21:16:55.879962 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:55.879938    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7a93e7f-28de-4f72-9ae2-15f8ca15be89-model-cache\") pod \"auth-disabled-test-kserve-77c4d7f586-tj7fp\" (UID: \"b7a93e7f-28de-4f72-9ae2-15f8ca15be89\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c4d7f586-tj7fp"
Sep 17 21:16:55.880017 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:55.880002    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7a93e7f-28de-4f72-9ae2-15f8ca15be89-tmp-dir\") pod \"auth-disabled-test-kserve-77c4d7f586-tj7fp\" (UID: \"b7a93e7f-28de-4f72-9ae2-15f8ca15be89\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c4d7f586-tj7fp"
Sep 17 21:16:55.880488 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:55.880464    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b7a93e7f-28de-4f72-9ae2-15f8ca15be89-cabundle-cert\") pod \"auth-disabled-test-kserve-77c4d7f586-tj7fp\" (UID: \"b7a93e7f-28de-4f72-9ae2-15f8ca15be89\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c4d7f586-tj7fp"
Sep 17 21:16:55.881726 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:55.881705    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7a93e7f-28de-4f72-9ae2-15f8ca15be89-dshm\") pod \"auth-disabled-test-kserve-77c4d7f586-tj7fp\" (UID: \"b7a93e7f-28de-4f72-9ae2-15f8ca15be89\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c4d7f586-tj7fp"
Sep 17 21:16:55.882022 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:55.882003    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7a93e7f-28de-4f72-9ae2-15f8ca15be89-tls-certs\") pod \"auth-disabled-test-kserve-77c4d7f586-tj7fp\" (UID: \"b7a93e7f-28de-4f72-9ae2-15f8ca15be89\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c4d7f586-tj7fp"
Sep 17 21:16:55.888640 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:55.888623    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq4nt\" (UniqueName: \"kubernetes.io/projected/b7a93e7f-28de-4f72-9ae2-15f8ca15be89-kube-api-access-lq4nt\") pod \"auth-disabled-test-kserve-77c4d7f586-tj7fp\" (UID: \"b7a93e7f-28de-4f72-9ae2-15f8ca15be89\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c4d7f586-tj7fp"
Sep 17 21:16:56.030915 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:56.030803    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:16:56.070040 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:56.070014    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c4d7f586-tj7fp"
Sep 17 21:16:56.354939 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:56.354831    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c4d7f586-tj7fp"]
Sep 17 21:16:56.359472 ip-10-0-139-36 kubenswrapper[2575]: W0917 21:16:56.359431    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7a93e7f_28de_4f72_9ae2_15f8ca15be89.slice/crio-63a4b1289f508f2800ccaa39a33e28865deeeafb914ab3e8a1b431e667b1d4c8 WatchSource:0}: Error finding container 63a4b1289f508f2800ccaa39a33e28865deeeafb914ab3e8a1b431e667b1d4c8: Status 404 returned error can't find the container with id 63a4b1289f508f2800ccaa39a33e28865deeeafb914ab3e8a1b431e667b1d4c8
Sep 17 21:16:56.477313 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:56.477267    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c4d7f586-tj7fp" event={"ID":"b7a93e7f-28de-4f72-9ae2-15f8ca15be89","Type":"ContainerStarted","Data":"0b79193c95119adedaa01c1a075f7028b9fc390c14ff7f0c040035e681b73a0f"}
Sep 17 21:16:56.477511 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:56.477332    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c4d7f586-tj7fp" event={"ID":"b7a93e7f-28de-4f72-9ae2-15f8ca15be89","Type":"ContainerStarted","Data":"63a4b1289f508f2800ccaa39a33e28865deeeafb914ab3e8a1b431e667b1d4c8"}
Sep 17 21:16:59.493397 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:59.493356    2575 generic.go:356] "Generic (PLEG): container finished" podID="b7a93e7f-28de-4f72-9ae2-15f8ca15be89" containerID="0b79193c95119adedaa01c1a075f7028b9fc390c14ff7f0c040035e681b73a0f" exitCode=0
Sep 17 21:16:59.493781 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:16:59.493436    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c4d7f586-tj7fp" event={"ID":"b7a93e7f-28de-4f72-9ae2-15f8ca15be89","Type":"ContainerDied","Data":"0b79193c95119adedaa01c1a075f7028b9fc390c14ff7f0c040035e681b73a0f"}
Sep 17 21:17:00.500933 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:17:00.500897    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c4d7f586-tj7fp" event={"ID":"b7a93e7f-28de-4f72-9ae2-15f8ca15be89","Type":"ContainerStarted","Data":"2e79f469db1d56c1926493da6a9d6df9a191b8859c954ff623aea93b621d0f7f"}
Sep 17 21:17:00.530124 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:17:00.530065    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c4d7f586-tj7fp" podStartSLOduration=3.210238825 podStartE2EDuration="5.530043558s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.319804733s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:16:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:17:00.526228575 +0000 UTC m=+1685.678672953" watchObservedRunningTime="2026-09-17 21:17:00.530043558 +0000 UTC m=+1685.682487935"
Sep 17 21:17:01.070477 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:17:01.070433    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c4d7f586-tj7fp"
Sep 17 21:17:03.029749 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:17:03.029711    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:17:06.070704 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:17:06.070661    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c4d7f586-tj7fp"
Sep 17 21:17:06.072568 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:17:06.072534    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c4d7f586-tj7fp" podUID="b7a93e7f-28de-4f72-9ae2-15f8ca15be89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Sep 17 21:17:14.753506 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:17:14.753465    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:17:14.841911 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:17:14.841868    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:17:16.070865 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:17:16.070802    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c4d7f586-tj7fp" podUID="b7a93e7f-28de-4f72-9ae2-15f8ca15be89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Sep 17 21:17:26.071207 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:17:26.071112    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c4d7f586-tj7fp" podUID="b7a93e7f-28de-4f72-9ae2-15f8ca15be89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Sep 17 21:17:36.071212 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:17:36.071165    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c4d7f586-tj7fp" podUID="b7a93e7f-28de-4f72-9ae2-15f8ca15be89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Sep 17 21:17:46.071354 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:17:46.071304    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c4d7f586-tj7fp" podUID="b7a93e7f-28de-4f72-9ae2-15f8ca15be89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Sep 17 21:17:56.071158 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:17:56.071108    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c4d7f586-tj7fp" podUID="b7a93e7f-28de-4f72-9ae2-15f8ca15be89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Sep 17 21:18:04.508046 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:04.508005    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:18:04.605989 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:04.605949    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:18:06.071158 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:06.071106    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c4d7f586-tj7fp" podUID="b7a93e7f-28de-4f72-9ae2-15f8ca15be89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Sep 17 21:18:11.063405 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:11.063364    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:18:11.315911 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:11.315770    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:18:16.071185 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:16.071128    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c4d7f586-tj7fp" podUID="b7a93e7f-28de-4f72-9ae2-15f8ca15be89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Sep 17 21:18:17.439988 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:17.439947    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:18:17.558656 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:17.558616    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:18:18.498196 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:18.498158    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q"]
Sep 17 21:18:18.502605 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:18.502583    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q"
Sep 17 21:18:18.511115 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:18.511089    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 21:18:18.511286 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:18.511261    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Sep 17 21:18:18.511563 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:18.511542    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 17 21:18:18.512026 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:18.512006    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 17 21:18:18.512128 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:18.512049    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 17 21:18:18.512345 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:18.512327    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-7lbzg\""
Sep 17 21:18:18.529960 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:18.529929    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q"]
Sep 17 21:18:18.546937 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:18.546903    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bdf58fe-c601-4974-b10f-c028053f81e6-home\") pod \"tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q\" (UID: \"9bdf58fe-c601-4974-b10f-c028053f81e6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q"
Sep 17 21:18:18.547204 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:18.546955    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9bdf58fe-c601-4974-b10f-c028053f81e6-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q\" (UID: \"9bdf58fe-c601-4974-b10f-c028053f81e6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q"
Sep 17 21:18:18.547204 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:18.546987    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bdf58fe-c601-4974-b10f-c028053f81e6-tls-certs\") pod \"tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q\" (UID: \"9bdf58fe-c601-4974-b10f-c028053f81e6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q"
Sep 17 21:18:18.547204 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:18.547022    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bdf58fe-c601-4974-b10f-c028053f81e6-tmp-dir\") pod \"tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q\" (UID: \"9bdf58fe-c601-4974-b10f-c028053f81e6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q"
Sep 17 21:18:18.547204 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:18.547180    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9bdf58fe-c601-4974-b10f-c028053f81e6-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q\" (UID: \"9bdf58fe-c601-4974-b10f-c028053f81e6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q"
Sep 17 21:18:18.547482 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:18.547237    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2jtv\" (UniqueName: \"kubernetes.io/projected/9bdf58fe-c601-4974-b10f-c028053f81e6-kube-api-access-j2jtv\") pod \"tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q\" (UID: \"9bdf58fe-c601-4974-b10f-c028053f81e6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q"
Sep 17 21:18:18.547482 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:18.547311    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bdf58fe-c601-4974-b10f-c028053f81e6-dshm\") pod \"tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q\" (UID: \"9bdf58fe-c601-4974-b10f-c028053f81e6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q"
Sep 17 21:18:18.547482 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:18.547355    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bdf58fe-c601-4974-b10f-c028053f81e6-model-cache\") pod \"tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q\" (UID: \"9bdf58fe-c601-4974-b10f-c028053f81e6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q"
Sep 17 21:18:18.547640 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:18.547501    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:18:18.648587 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:18.648552    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9bdf58fe-c601-4974-b10f-c028053f81e6-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q\" (UID: \"9bdf58fe-c601-4974-b10f-c028053f81e6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q"
Sep 17 21:18:18.648587 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:18.648587    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2jtv\" (UniqueName: \"kubernetes.io/projected/9bdf58fe-c601-4974-b10f-c028053f81e6-kube-api-access-j2jtv\") pod \"tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q\" (UID: \"9bdf58fe-c601-4974-b10f-c028053f81e6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q"
Sep 17 21:18:18.648861 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:18.648618    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bdf58fe-c601-4974-b10f-c028053f81e6-dshm\") pod \"tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q\" (UID: \"9bdf58fe-c601-4974-b10f-c028053f81e6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q"
Sep 17 21:18:18.648861 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:18.648658    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bdf58fe-c601-4974-b10f-c028053f81e6-model-cache\") pod \"tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q\" (UID: \"9bdf58fe-c601-4974-b10f-c028053f81e6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q"
Sep 17 21:18:18.648861 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:18.648677    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bdf58fe-c601-4974-b10f-c028053f81e6-home\") pod \"tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q\" (UID: \"9bdf58fe-c601-4974-b10f-c028053f81e6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q"
Sep 17 21:18:18.648861 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:18.648703    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9bdf58fe-c601-4974-b10f-c028053f81e6-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q\" (UID: \"9bdf58fe-c601-4974-b10f-c028053f81e6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q"
Sep 17 21:18:18.648861 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:18.648721    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bdf58fe-c601-4974-b10f-c028053f81e6-tls-certs\") pod \"tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q\" (UID: \"9bdf58fe-c601-4974-b10f-c028053f81e6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q"
Sep 17 21:18:18.648861 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:18.648770    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bdf58fe-c601-4974-b10f-c028053f81e6-tmp-dir\") pod \"tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q\" (UID: \"9bdf58fe-c601-4974-b10f-c028053f81e6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q"
Sep 17 21:18:18.649182 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:18.649144    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9bdf58fe-c601-4974-b10f-c028053f81e6-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q\" (UID: \"9bdf58fe-c601-4974-b10f-c028053f81e6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q"
Sep 17 21:18:18.649257 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:18.649234    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bdf58fe-c601-4974-b10f-c028053f81e6-model-cache\") pod \"tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q\" (UID: \"9bdf58fe-c601-4974-b10f-c028053f81e6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q"
Sep 17 21:18:18.649320 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:18.649260    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bdf58fe-c601-4974-b10f-c028053f81e6-home\") pod \"tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q\" (UID: \"9bdf58fe-c601-4974-b10f-c028053f81e6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q"
Sep 17 21:18:18.649320 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:18.649286    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bdf58fe-c601-4974-b10f-c028053f81e6-tmp-dir\") pod \"tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q\" (UID: \"9bdf58fe-c601-4974-b10f-c028053f81e6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q"
Sep 17 21:18:18.649573 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:18.649521    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9bdf58fe-c601-4974-b10f-c028053f81e6-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q\" (UID: \"9bdf58fe-c601-4974-b10f-c028053f81e6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q"
Sep 17 21:18:18.651402 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:18.651373    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bdf58fe-c601-4974-b10f-c028053f81e6-dshm\") pod \"tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q\" (UID: \"9bdf58fe-c601-4974-b10f-c028053f81e6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q"
Sep 17 21:18:18.651513 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:18.651497    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bdf58fe-c601-4974-b10f-c028053f81e6-tls-certs\") pod \"tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q\" (UID: \"9bdf58fe-c601-4974-b10f-c028053f81e6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q"
Sep 17 21:18:18.659700 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:18.659677    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2jtv\" (UniqueName: \"kubernetes.io/projected/9bdf58fe-c601-4974-b10f-c028053f81e6-kube-api-access-j2jtv\") pod \"tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q\" (UID: \"9bdf58fe-c601-4974-b10f-c028053f81e6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q"
Sep 17 21:18:18.813879 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:18.813781    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q"
Sep 17 21:18:18.816322 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:18.816288    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:18:18.980041 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:18.979996    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:18:19.095590 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:19.095502    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q"]
Sep 17 21:18:19.100740 ip-10-0-139-36 kubenswrapper[2575]: W0917 21:18:19.100711    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bdf58fe_c601_4974_b10f_c028053f81e6.slice/crio-51a9d798a7e81b62b2e01d6bb190c9725e99d90b78e0007595f22ecec1ab39e1 WatchSource:0}: Error finding container 51a9d798a7e81b62b2e01d6bb190c9725e99d90b78e0007595f22ecec1ab39e1: Status 404 returned error can't find the container with id 51a9d798a7e81b62b2e01d6bb190c9725e99d90b78e0007595f22ecec1ab39e1
Sep 17 21:18:19.926177 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:19.926139    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q" event={"ID":"9bdf58fe-c601-4974-b10f-c028053f81e6","Type":"ContainerStarted","Data":"9aaed60a82d15db9d44a14988e75a66ee7c963c1c4e4e48116f328bcc98c2d8a"}
Sep 17 21:18:19.926633 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:19.926219    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q" event={"ID":"9bdf58fe-c601-4974-b10f-c028053f81e6","Type":"ContainerStarted","Data":"51a9d798a7e81b62b2e01d6bb190c9725e99d90b78e0007595f22ecec1ab39e1"}
Sep 17 21:18:21.940079 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:21.939979    2575 generic.go:356] "Generic (PLEG): container finished" podID="9bdf58fe-c601-4974-b10f-c028053f81e6" containerID="9aaed60a82d15db9d44a14988e75a66ee7c963c1c4e4e48116f328bcc98c2d8a" exitCode=0
Sep 17 21:18:21.940079 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:21.940051    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q" event={"ID":"9bdf58fe-c601-4974-b10f-c028053f81e6","Type":"ContainerDied","Data":"9aaed60a82d15db9d44a14988e75a66ee7c963c1c4e4e48116f328bcc98c2d8a"}
Sep 17 21:18:22.946740 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:22.946706    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q" event={"ID":"9bdf58fe-c601-4974-b10f-c028053f81e6","Type":"ContainerStarted","Data":"704f9bfcc85bb7fd296b47e9641a36e5ab645cc12e87cc833018b180904b9028"}
Sep 17 21:18:22.969393 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:22.969337    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q" podStartSLOduration=2.604345809 podStartE2EDuration="4.969320144s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.364974335s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:18:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:18:22.967610765 +0000 UTC m=+1768.120055142" watchObservedRunningTime="2026-09-17 21:18:22.969320144 +0000 UTC m=+1768.121764519"
Sep 17 21:18:23.814190 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:23.814152    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q"
Sep 17 21:18:26.070676 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:26.070633    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c4d7f586-tj7fp" podUID="b7a93e7f-28de-4f72-9ae2-15f8ca15be89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Sep 17 21:18:28.814866 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:28.814810    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q"
Sep 17 21:18:28.828062 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:28.828037    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q"
Sep 17 21:18:28.989381 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:28.989348    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q"
Sep 17 21:18:36.071279 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:36.071240    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c4d7f586-tj7fp" podUID="b7a93e7f-28de-4f72-9ae2-15f8ca15be89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Sep 17 21:18:36.536601 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:36.536531    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:18:46.070470 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:46.070421    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c4d7f586-tj7fp" podUID="b7a93e7f-28de-4f72-9ae2-15f8ca15be89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Sep 17 21:18:56.080597 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:56.080567    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c4d7f586-tj7fp"
Sep 17 21:18:56.088509 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:56.088484    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c4d7f586-tj7fp"
Sep 17 21:18:57.754464 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:57.754428    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q"]
Sep 17 21:18:57.761571 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:57.761543    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:18:57.765352 ip-10-0-139-36 kubenswrapper[2575]: E0917 21:18:57.765321    2575 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 21:18:57.765352 ip-10-0-139-36 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:18:57.765352 ip-10-0-139-36 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 17 21:18:57.765352 ip-10-0-139-36 kubenswrapper[2575]:  > containerID="704f9bfcc85bb7fd296b47e9641a36e5ab645cc12e87cc833018b180904b9028" cmd=["/bin/sleep","15"]
Sep 17 21:18:57.765633 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:57.765375    2575 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q" message="" err=<
Sep 17 21:18:57.765633 ip-10-0-139-36 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:18:57.765633 ip-10-0-139-36 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 17 21:18:57.765633 ip-10-0-139-36 kubenswrapper[2575]:  >
Sep 17 21:18:57.765633 ip-10-0-139-36 kubenswrapper[2575]: E0917 21:18:57.765392    2575 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 21:18:57.765633 ip-10-0-139-36 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:18:57.765633 ip-10-0-139-36 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 17 21:18:57.765633 ip-10-0-139-36 kubenswrapper[2575]:  > pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q" podUID="9bdf58fe-c601-4974-b10f-c028053f81e6" containerName="main" containerID="cri-o://704f9bfcc85bb7fd296b47e9641a36e5ab645cc12e87cc833018b180904b9028"
Sep 17 21:18:57.765633 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:57.765434    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q" podUID="9bdf58fe-c601-4974-b10f-c028053f81e6" containerName="main" containerID="cri-o://704f9bfcc85bb7fd296b47e9641a36e5ab645cc12e87cc833018b180904b9028" gracePeriod=60
Sep 17 21:18:57.839071 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:57.839036    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:18:57.978354 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:57.978239    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q" podUID="9bdf58fe-c601-4974-b10f-c028053f81e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Sep 17 21:18:58.116411 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:58.116386    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q"
Sep 17 21:18:58.120526 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:58.120502    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bdf58fe-c601-4974-b10f-c028053f81e6-dshm\") pod \"9bdf58fe-c601-4974-b10f-c028053f81e6\" (UID: \"9bdf58fe-c601-4974-b10f-c028053f81e6\") "
Sep 17 21:18:58.120644 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:58.120554    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bdf58fe-c601-4974-b10f-c028053f81e6-tmp-dir\") pod \"9bdf58fe-c601-4974-b10f-c028053f81e6\" (UID: \"9bdf58fe-c601-4974-b10f-c028053f81e6\") "
Sep 17 21:18:58.120644 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:58.120590    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bdf58fe-c601-4974-b10f-c028053f81e6-home\") pod \"9bdf58fe-c601-4974-b10f-c028053f81e6\" (UID: \"9bdf58fe-c601-4974-b10f-c028053f81e6\") "
Sep 17 21:18:58.120644 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:58.120610    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9bdf58fe-c601-4974-b10f-c028053f81e6-cabundle-cert\") pod \"9bdf58fe-c601-4974-b10f-c028053f81e6\" (UID: \"9bdf58fe-c601-4974-b10f-c028053f81e6\") "
Sep 17 21:18:58.120644 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:58.120630    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j2jtv\" (UniqueName: \"kubernetes.io/projected/9bdf58fe-c601-4974-b10f-c028053f81e6-kube-api-access-j2jtv\") pod \"9bdf58fe-c601-4974-b10f-c028053f81e6\" (UID: \"9bdf58fe-c601-4974-b10f-c028053f81e6\") "
Sep 17 21:18:58.120893 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:58.120681    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bdf58fe-c601-4974-b10f-c028053f81e6-model-cache\") pod \"9bdf58fe-c601-4974-b10f-c028053f81e6\" (UID: \"9bdf58fe-c601-4974-b10f-c028053f81e6\") "
Sep 17 21:18:58.120893 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:58.120738    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bdf58fe-c601-4974-b10f-c028053f81e6-tls-certs\") pod \"9bdf58fe-c601-4974-b10f-c028053f81e6\" (UID: \"9bdf58fe-c601-4974-b10f-c028053f81e6\") "
Sep 17 21:18:58.120893 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:58.120778    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9bdf58fe-c601-4974-b10f-c028053f81e6-kserve-provision-location\") pod \"9bdf58fe-c601-4974-b10f-c028053f81e6\" (UID: \"9bdf58fe-c601-4974-b10f-c028053f81e6\") "
Sep 17 21:18:58.120893 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:58.120842    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bdf58fe-c601-4974-b10f-c028053f81e6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9bdf58fe-c601-4974-b10f-c028053f81e6" (UID: "9bdf58fe-c601-4974-b10f-c028053f81e6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:18:58.120893 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:58.120880    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bdf58fe-c601-4974-b10f-c028053f81e6-home" (OuterVolumeSpecName: "home") pod "9bdf58fe-c601-4974-b10f-c028053f81e6" (UID: "9bdf58fe-c601-4974-b10f-c028053f81e6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:18:58.121162 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:58.120969    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bdf58fe-c601-4974-b10f-c028053f81e6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9bdf58fe-c601-4974-b10f-c028053f81e6" (UID: "9bdf58fe-c601-4974-b10f-c028053f81e6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:18:58.121162 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:58.121023    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bdf58fe-c601-4974-b10f-c028053f81e6-model-cache" (OuterVolumeSpecName: "model-cache") pod "9bdf58fe-c601-4974-b10f-c028053f81e6" (UID: "9bdf58fe-c601-4974-b10f-c028053f81e6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:18:58.121162 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:58.121117    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bdf58fe-c601-4974-b10f-c028053f81e6-home\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:18:58.121162 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:58.121135    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9bdf58fe-c601-4974-b10f-c028053f81e6-cabundle-cert\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:18:58.121162 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:58.121148    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bdf58fe-c601-4974-b10f-c028053f81e6-model-cache\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:18:58.121162 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:58.121160    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bdf58fe-c601-4974-b10f-c028053f81e6-tmp-dir\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:18:58.123349 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:58.123312    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bdf58fe-c601-4974-b10f-c028053f81e6-kube-api-access-j2jtv" (OuterVolumeSpecName: "kube-api-access-j2jtv") pod "9bdf58fe-c601-4974-b10f-c028053f81e6" (UID: "9bdf58fe-c601-4974-b10f-c028053f81e6"). InnerVolumeSpecName "kube-api-access-j2jtv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:18:58.123464 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:58.123372    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bdf58fe-c601-4974-b10f-c028053f81e6-dshm" (OuterVolumeSpecName: "dshm") pod "9bdf58fe-c601-4974-b10f-c028053f81e6" (UID: "9bdf58fe-c601-4974-b10f-c028053f81e6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:18:58.123464 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:58.123388    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bdf58fe-c601-4974-b10f-c028053f81e6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9bdf58fe-c601-4974-b10f-c028053f81e6" (UID: "9bdf58fe-c601-4974-b10f-c028053f81e6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:18:58.132481 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:58.132456    2575 generic.go:356] "Generic (PLEG): container finished" podID="9bdf58fe-c601-4974-b10f-c028053f81e6" containerID="704f9bfcc85bb7fd296b47e9641a36e5ab645cc12e87cc833018b180904b9028" exitCode=0
Sep 17 21:18:58.132594 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:58.132517    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q"
Sep 17 21:18:58.132594 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:58.132538    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q" event={"ID":"9bdf58fe-c601-4974-b10f-c028053f81e6","Type":"ContainerDied","Data":"704f9bfcc85bb7fd296b47e9641a36e5ab645cc12e87cc833018b180904b9028"}
Sep 17 21:18:58.132714 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:58.132598    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q" event={"ID":"9bdf58fe-c601-4974-b10f-c028053f81e6","Type":"ContainerDied","Data":"51a9d798a7e81b62b2e01d6bb190c9725e99d90b78e0007595f22ecec1ab39e1"}
Sep 17 21:18:58.132714 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:58.132630    2575 scope.go:117] "RemoveContainer" containerID="704f9bfcc85bb7fd296b47e9641a36e5ab645cc12e87cc833018b180904b9028"
Sep 17 21:18:58.142952 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:58.142925    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bdf58fe-c601-4974-b10f-c028053f81e6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9bdf58fe-c601-4974-b10f-c028053f81e6" (UID: "9bdf58fe-c601-4974-b10f-c028053f81e6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:18:58.143045 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:58.142951    2575 scope.go:117] "RemoveContainer" containerID="9aaed60a82d15db9d44a14988e75a66ee7c963c1c4e4e48116f328bcc98c2d8a"
Sep 17 21:18:58.153115 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:58.153091    2575 scope.go:117] "RemoveContainer" containerID="704f9bfcc85bb7fd296b47e9641a36e5ab645cc12e87cc833018b180904b9028"
Sep 17 21:18:58.153445 ip-10-0-139-36 kubenswrapper[2575]: E0917 21:18:58.153419    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"704f9bfcc85bb7fd296b47e9641a36e5ab645cc12e87cc833018b180904b9028\": container with ID starting with 704f9bfcc85bb7fd296b47e9641a36e5ab645cc12e87cc833018b180904b9028 not found: ID does not exist" containerID="704f9bfcc85bb7fd296b47e9641a36e5ab645cc12e87cc833018b180904b9028"
Sep 17 21:18:58.153533 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:58.153456    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"704f9bfcc85bb7fd296b47e9641a36e5ab645cc12e87cc833018b180904b9028"} err="failed to get container status \"704f9bfcc85bb7fd296b47e9641a36e5ab645cc12e87cc833018b180904b9028\": rpc error: code = NotFound desc = could not find container \"704f9bfcc85bb7fd296b47e9641a36e5ab645cc12e87cc833018b180904b9028\": container with ID starting with 704f9bfcc85bb7fd296b47e9641a36e5ab645cc12e87cc833018b180904b9028 not found: ID does not exist"
Sep 17 21:18:58.153533 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:58.153480    2575 scope.go:117] "RemoveContainer" containerID="9aaed60a82d15db9d44a14988e75a66ee7c963c1c4e4e48116f328bcc98c2d8a"
Sep 17 21:18:58.153758 ip-10-0-139-36 kubenswrapper[2575]: E0917 21:18:58.153736    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9aaed60a82d15db9d44a14988e75a66ee7c963c1c4e4e48116f328bcc98c2d8a\": container with ID starting with 9aaed60a82d15db9d44a14988e75a66ee7c963c1c4e4e48116f328bcc98c2d8a not found: ID does not exist" containerID="9aaed60a82d15db9d44a14988e75a66ee7c963c1c4e4e48116f328bcc98c2d8a"
Sep 17 21:18:58.153825 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:58.153762    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9aaed60a82d15db9d44a14988e75a66ee7c963c1c4e4e48116f328bcc98c2d8a"} err="failed to get container status \"9aaed60a82d15db9d44a14988e75a66ee7c963c1c4e4e48116f328bcc98c2d8a\": rpc error: code = NotFound desc = could not find container \"9aaed60a82d15db9d44a14988e75a66ee7c963c1c4e4e48116f328bcc98c2d8a\": container with ID starting with 9aaed60a82d15db9d44a14988e75a66ee7c963c1c4e4e48116f328bcc98c2d8a not found: ID does not exist"
Sep 17 21:18:58.221719 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:58.221677    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bdf58fe-c601-4974-b10f-c028053f81e6-tls-certs\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:18:58.221719 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:58.221705    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9bdf58fe-c601-4974-b10f-c028053f81e6-kserve-provision-location\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:18:58.221719 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:58.221713    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bdf58fe-c601-4974-b10f-c028053f81e6-dshm\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:18:58.221719 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:58.221723    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2jtv\" (UniqueName: \"kubernetes.io/projected/9bdf58fe-c601-4974-b10f-c028053f81e6-kube-api-access-j2jtv\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:18:58.472035 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:58.472003    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q"]
Sep 17 21:18:58.477589 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:58.477564    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-pqp4q"]
Sep 17 21:18:59.357944 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:18:59.357903    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9bdf58fe-c601-4974-b10f-c028053f81e6" path="/var/lib/kubelet/pods/9bdf58fe-c601-4974-b10f-c028053f81e6/volumes"
Sep 17 21:19:01.628035 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:19:01.627996    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c4d7f586-tj7fp"]
Sep 17 21:19:01.631488 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:19:01.631455    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:19:01.702113 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:19:01.702070    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:19:16.652047 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:19:16.652007    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c4d7f586-tj7fp" podUID="b7a93e7f-28de-4f72-9ae2-15f8ca15be89" containerName="main" containerID="cri-o://2e79f469db1d56c1926493da6a9d6df9a191b8859c954ff623aea93b621d0f7f" gracePeriod=45
Sep 17 21:19:17.081538 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:19:17.081499    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c4d7f586-tj7fp" podUID="b7a93e7f-28de-4f72-9ae2-15f8ca15be89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Sep 17 21:19:17.236632 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:19:17.236597    2575 generic.go:356] "Generic (PLEG): container finished" podID="b7a93e7f-28de-4f72-9ae2-15f8ca15be89" containerID="2e79f469db1d56c1926493da6a9d6df9a191b8859c954ff623aea93b621d0f7f" exitCode=0
Sep 17 21:19:17.236801 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:19:17.236667    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c4d7f586-tj7fp" event={"ID":"b7a93e7f-28de-4f72-9ae2-15f8ca15be89","Type":"ContainerDied","Data":"2e79f469db1d56c1926493da6a9d6df9a191b8859c954ff623aea93b621d0f7f"}
Sep 17 21:19:17.324535 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:19:17.324509    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c4d7f586-tj7fp"
Sep 17 21:19:17.394261 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:19:17.394160    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7a93e7f-28de-4f72-9ae2-15f8ca15be89-model-cache\") pod \"b7a93e7f-28de-4f72-9ae2-15f8ca15be89\" (UID: \"b7a93e7f-28de-4f72-9ae2-15f8ca15be89\") "
Sep 17 21:19:17.394261 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:19:17.394232    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b7a93e7f-28de-4f72-9ae2-15f8ca15be89-cabundle-cert\") pod \"b7a93e7f-28de-4f72-9ae2-15f8ca15be89\" (UID: \"b7a93e7f-28de-4f72-9ae2-15f8ca15be89\") "
Sep 17 21:19:17.394495 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:19:17.394271    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7a93e7f-28de-4f72-9ae2-15f8ca15be89-dshm\") pod \"b7a93e7f-28de-4f72-9ae2-15f8ca15be89\" (UID: \"b7a93e7f-28de-4f72-9ae2-15f8ca15be89\") "
Sep 17 21:19:17.394495 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:19:17.394304    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7a93e7f-28de-4f72-9ae2-15f8ca15be89-home\") pod \"b7a93e7f-28de-4f72-9ae2-15f8ca15be89\" (UID: \"b7a93e7f-28de-4f72-9ae2-15f8ca15be89\") "
Sep 17 21:19:17.394495 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:19:17.394362    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7a93e7f-28de-4f72-9ae2-15f8ca15be89-tmp-dir\") pod \"b7a93e7f-28de-4f72-9ae2-15f8ca15be89\" (UID: \"b7a93e7f-28de-4f72-9ae2-15f8ca15be89\") "
Sep 17 21:19:17.394495 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:19:17.394389    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b7a93e7f-28de-4f72-9ae2-15f8ca15be89-kserve-provision-location\") pod \"b7a93e7f-28de-4f72-9ae2-15f8ca15be89\" (UID: \"b7a93e7f-28de-4f72-9ae2-15f8ca15be89\") "
Sep 17 21:19:17.394495 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:19:17.394432    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7a93e7f-28de-4f72-9ae2-15f8ca15be89-tls-certs\") pod \"b7a93e7f-28de-4f72-9ae2-15f8ca15be89\" (UID: \"b7a93e7f-28de-4f72-9ae2-15f8ca15be89\") "
Sep 17 21:19:17.394495 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:19:17.394444    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7a93e7f-28de-4f72-9ae2-15f8ca15be89-model-cache" (OuterVolumeSpecName: "model-cache") pod "b7a93e7f-28de-4f72-9ae2-15f8ca15be89" (UID: "b7a93e7f-28de-4f72-9ae2-15f8ca15be89"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:19:17.394813 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:19:17.394525    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lq4nt\" (UniqueName: \"kubernetes.io/projected/b7a93e7f-28de-4f72-9ae2-15f8ca15be89-kube-api-access-lq4nt\") pod \"b7a93e7f-28de-4f72-9ae2-15f8ca15be89\" (UID: \"b7a93e7f-28de-4f72-9ae2-15f8ca15be89\") "
Sep 17 21:19:17.394813 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:19:17.394745    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7a93e7f-28de-4f72-9ae2-15f8ca15be89-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b7a93e7f-28de-4f72-9ae2-15f8ca15be89" (UID: "b7a93e7f-28de-4f72-9ae2-15f8ca15be89"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:19:17.394813 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:19:17.394807    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7a93e7f-28de-4f72-9ae2-15f8ca15be89-model-cache\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:19:17.397064 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:19:17.397022    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7a93e7f-28de-4f72-9ae2-15f8ca15be89-dshm" (OuterVolumeSpecName: "dshm") pod "b7a93e7f-28de-4f72-9ae2-15f8ca15be89" (UID: "b7a93e7f-28de-4f72-9ae2-15f8ca15be89"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:19:17.397191 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:19:17.397067    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7a93e7f-28de-4f72-9ae2-15f8ca15be89-kube-api-access-lq4nt" (OuterVolumeSpecName: "kube-api-access-lq4nt") pod "b7a93e7f-28de-4f72-9ae2-15f8ca15be89" (UID: "b7a93e7f-28de-4f72-9ae2-15f8ca15be89"). InnerVolumeSpecName "kube-api-access-lq4nt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:19:17.397191 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:19:17.397071    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7a93e7f-28de-4f72-9ae2-15f8ca15be89-home" (OuterVolumeSpecName: "home") pod "b7a93e7f-28de-4f72-9ae2-15f8ca15be89" (UID: "b7a93e7f-28de-4f72-9ae2-15f8ca15be89"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:19:17.397360 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:19:17.397334    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7a93e7f-28de-4f72-9ae2-15f8ca15be89-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b7a93e7f-28de-4f72-9ae2-15f8ca15be89" (UID: "b7a93e7f-28de-4f72-9ae2-15f8ca15be89"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:19:17.407264 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:19:17.407219    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7a93e7f-28de-4f72-9ae2-15f8ca15be89-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b7a93e7f-28de-4f72-9ae2-15f8ca15be89" (UID: "b7a93e7f-28de-4f72-9ae2-15f8ca15be89"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:19:17.418219 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:19:17.418182    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7a93e7f-28de-4f72-9ae2-15f8ca15be89-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b7a93e7f-28de-4f72-9ae2-15f8ca15be89" (UID: "b7a93e7f-28de-4f72-9ae2-15f8ca15be89"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:19:17.495744 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:19:17.495712    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7a93e7f-28de-4f72-9ae2-15f8ca15be89-tmp-dir\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:19:17.495744 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:19:17.495740    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b7a93e7f-28de-4f72-9ae2-15f8ca15be89-kserve-provision-location\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:19:17.495744 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:19:17.495752    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7a93e7f-28de-4f72-9ae2-15f8ca15be89-tls-certs\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:19:17.496012 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:19:17.495762    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lq4nt\" (UniqueName: \"kubernetes.io/projected/b7a93e7f-28de-4f72-9ae2-15f8ca15be89-kube-api-access-lq4nt\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:19:17.496012 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:19:17.495771    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b7a93e7f-28de-4f72-9ae2-15f8ca15be89-cabundle-cert\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:19:17.496012 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:19:17.495779    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7a93e7f-28de-4f72-9ae2-15f8ca15be89-dshm\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:19:17.496012 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:19:17.495786    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7a93e7f-28de-4f72-9ae2-15f8ca15be89-home\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:19:18.243692 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:19:18.243662    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c4d7f586-tj7fp"
Sep 17 21:19:18.244181 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:19:18.243660    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c4d7f586-tj7fp" event={"ID":"b7a93e7f-28de-4f72-9ae2-15f8ca15be89","Type":"ContainerDied","Data":"63a4b1289f508f2800ccaa39a33e28865deeeafb914ab3e8a1b431e667b1d4c8"}
Sep 17 21:19:18.244181 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:19:18.243794    2575 scope.go:117] "RemoveContainer" containerID="2e79f469db1d56c1926493da6a9d6df9a191b8859c954ff623aea93b621d0f7f"
Sep 17 21:19:18.253158 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:19:18.253138    2575 scope.go:117] "RemoveContainer" containerID="0b79193c95119adedaa01c1a075f7028b9fc390c14ff7f0c040035e681b73a0f"
Sep 17 21:19:18.266323 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:19:18.266292    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c4d7f586-tj7fp"]
Sep 17 21:19:18.271348 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:19:18.271325    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c4d7f586-tj7fp"]
Sep 17 21:19:19.357976 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:19:19.357947    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7a93e7f-28de-4f72-9ae2-15f8ca15be89" path="/var/lib/kubelet/pods/b7a93e7f-28de-4f72-9ae2-15f8ca15be89/volumes"
Sep 17 21:19:27.065978 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:19:27.065938    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:19:27.154203 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:19:27.154175    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:19:46.687741 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:19:46.687706    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:19:46.865369 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:19:46.865327    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:19:51.880080 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:19:51.880043    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:19:51.973667 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:19:51.973636    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:19:52.184299 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:19:52.184187    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:19:52.284663 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:19:52.284627    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:19:53.985468 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:19:53.985424    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:19:55.513136 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:19:55.513076    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:19:55.842141 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:19:55.842050    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:19:56.213310 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:19:56.213276    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:19:59.207053 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:19:59.207019    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:20:00.173280 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:00.173248    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:20:00.247685 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:00.247640    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:20:05.105581 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:05.105532    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:20:05.298327 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:05.298291    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:20:26.191262 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:26.191156    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:20:36.126454 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:36.126420    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997hwqk4"]
Sep 17 21:20:36.126942 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:36.126799    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bdf58fe-c601-4974-b10f-c028053f81e6" containerName="main"
Sep 17 21:20:36.126942 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:36.126810    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bdf58fe-c601-4974-b10f-c028053f81e6" containerName="main"
Sep 17 21:20:36.126942 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:36.126819    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7a93e7f-28de-4f72-9ae2-15f8ca15be89" containerName="storage-initializer"
Sep 17 21:20:36.126942 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:36.126825    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7a93e7f-28de-4f72-9ae2-15f8ca15be89" containerName="storage-initializer"
Sep 17 21:20:36.126942 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:36.126832    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7a93e7f-28de-4f72-9ae2-15f8ca15be89" containerName="main"
Sep 17 21:20:36.126942 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:36.126838    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7a93e7f-28de-4f72-9ae2-15f8ca15be89" containerName="main"
Sep 17 21:20:36.126942 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:36.126864    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bdf58fe-c601-4974-b10f-c028053f81e6" containerName="storage-initializer"
Sep 17 21:20:36.126942 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:36.126870    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bdf58fe-c601-4974-b10f-c028053f81e6" containerName="storage-initializer"
Sep 17 21:20:36.126942 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:36.126943    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7a93e7f-28de-4f72-9ae2-15f8ca15be89" containerName="main"
Sep 17 21:20:36.127285 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:36.126954    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bdf58fe-c601-4974-b10f-c028053f81e6" containerName="main"
Sep 17 21:20:36.129867 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:36.129835    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997hwqk4"
Sep 17 21:20:36.135122 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:36.135077    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"default-dockercfg-znnc8\""
Sep 17 21:20:36.135122 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:36.135090    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs\""
Sep 17 21:20:36.135340 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:36.135111    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"kube-root-ca.crt\""
Sep 17 21:20:36.135340 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:36.135243    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-service-ca.crt\""
Sep 17 21:20:36.149681 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:36.149652    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997hwqk4"]
Sep 17 21:20:36.171472 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:36.171438    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e71f1d93-b032-41f9-bd45-3468ee381037-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997hwqk4\" (UID: \"e71f1d93-b032-41f9-bd45-3468ee381037\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997hwqk4"
Sep 17 21:20:36.171645 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:36.171478    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e71f1d93-b032-41f9-bd45-3468ee381037-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997hwqk4\" (UID: \"e71f1d93-b032-41f9-bd45-3468ee381037\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997hwqk4"
Sep 17 21:20:36.171645 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:36.171550    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e71f1d93-b032-41f9-bd45-3468ee381037-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997hwqk4\" (UID: \"e71f1d93-b032-41f9-bd45-3468ee381037\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997hwqk4"
Sep 17 21:20:36.171645 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:36.171567    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57ztx\" (UniqueName: \"kubernetes.io/projected/e71f1d93-b032-41f9-bd45-3468ee381037-kube-api-access-57ztx\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997hwqk4\" (UID: \"e71f1d93-b032-41f9-bd45-3468ee381037\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997hwqk4"
Sep 17 21:20:36.171832 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:36.171640    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e71f1d93-b032-41f9-bd45-3468ee381037-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997hwqk4\" (UID: \"e71f1d93-b032-41f9-bd45-3468ee381037\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997hwqk4"
Sep 17 21:20:36.171832 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:36.171737    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e71f1d93-b032-41f9-bd45-3468ee381037-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997hwqk4\" (UID: \"e71f1d93-b032-41f9-bd45-3468ee381037\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997hwqk4"
Sep 17 21:20:36.177168 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:36.177142    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:20:36.272825 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:36.272792    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e71f1d93-b032-41f9-bd45-3468ee381037-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997hwqk4\" (UID: \"e71f1d93-b032-41f9-bd45-3468ee381037\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997hwqk4"
Sep 17 21:20:36.273029 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:36.272836    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e71f1d93-b032-41f9-bd45-3468ee381037-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997hwqk4\" (UID: \"e71f1d93-b032-41f9-bd45-3468ee381037\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997hwqk4"
Sep 17 21:20:36.273029 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:36.272884    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e71f1d93-b032-41f9-bd45-3468ee381037-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997hwqk4\" (UID: \"e71f1d93-b032-41f9-bd45-3468ee381037\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997hwqk4"
Sep 17 21:20:36.273029 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:36.272921    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e71f1d93-b032-41f9-bd45-3468ee381037-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997hwqk4\" (UID: \"e71f1d93-b032-41f9-bd45-3468ee381037\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997hwqk4"
Sep 17 21:20:36.273029 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:36.272937    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57ztx\" (UniqueName: \"kubernetes.io/projected/e71f1d93-b032-41f9-bd45-3468ee381037-kube-api-access-57ztx\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997hwqk4\" (UID: \"e71f1d93-b032-41f9-bd45-3468ee381037\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997hwqk4"
Sep 17 21:20:36.273029 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:36.272972    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e71f1d93-b032-41f9-bd45-3468ee381037-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997hwqk4\" (UID: \"e71f1d93-b032-41f9-bd45-3468ee381037\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997hwqk4"
Sep 17 21:20:36.273343 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:36.273317    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e71f1d93-b032-41f9-bd45-3468ee381037-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997hwqk4\" (UID: \"e71f1d93-b032-41f9-bd45-3468ee381037\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997hwqk4"
Sep 17 21:20:36.273411 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:36.273351    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e71f1d93-b032-41f9-bd45-3468ee381037-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997hwqk4\" (UID: \"e71f1d93-b032-41f9-bd45-3468ee381037\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997hwqk4"
Sep 17 21:20:36.273649 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:36.273623    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e71f1d93-b032-41f9-bd45-3468ee381037-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997hwqk4\" (UID: \"e71f1d93-b032-41f9-bd45-3468ee381037\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997hwqk4"
Sep 17 21:20:36.275439 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:36.275417    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e71f1d93-b032-41f9-bd45-3468ee381037-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997hwqk4\" (UID: \"e71f1d93-b032-41f9-bd45-3468ee381037\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997hwqk4"
Sep 17 21:20:36.275676 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:36.275659    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e71f1d93-b032-41f9-bd45-3468ee381037-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997hwqk4\" (UID: \"e71f1d93-b032-41f9-bd45-3468ee381037\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997hwqk4"
Sep 17 21:20:36.282572 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:36.282552    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57ztx\" (UniqueName: \"kubernetes.io/projected/e71f1d93-b032-41f9-bd45-3468ee381037-kube-api-access-57ztx\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997hwqk4\" (UID: \"e71f1d93-b032-41f9-bd45-3468ee381037\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997hwqk4"
Sep 17 21:20:36.440276 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:36.440180    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997hwqk4"
Sep 17 21:20:36.732499 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:36.732465    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997hwqk4"]
Sep 17 21:20:36.733547 ip-10-0-139-36 kubenswrapper[2575]: W0917 21:20:36.733514    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode71f1d93_b032_41f9_bd45_3468ee381037.slice/crio-e10a2a8b35a9b15826956c7eacbbe83f151440b2f1fcd81cbff3dd4c156d0336 WatchSource:0}: Error finding container e10a2a8b35a9b15826956c7eacbbe83f151440b2f1fcd81cbff3dd4c156d0336: Status 404 returned error can't find the container with id e10a2a8b35a9b15826956c7eacbbe83f151440b2f1fcd81cbff3dd4c156d0336
Sep 17 21:20:37.624470 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:37.624439    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997hwqk4" event={"ID":"e71f1d93-b032-41f9-bd45-3468ee381037","Type":"ContainerStarted","Data":"d92862d3f66c9d11f20bf8ec7a7a71213d9aaa76b5e9492d773e399d2aa0aae1"}
Sep 17 21:20:37.624897 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:37.624489    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997hwqk4" event={"ID":"e71f1d93-b032-41f9-bd45-3468ee381037","Type":"ContainerStarted","Data":"e10a2a8b35a9b15826956c7eacbbe83f151440b2f1fcd81cbff3dd4c156d0336"}
Sep 17 21:20:37.644381 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:37.644332    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997hwqk4" podStartSLOduration=1.644314091 podStartE2EDuration="1.644314091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:20:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:20:37.642887476 +0000 UTC m=+1902.795331854" watchObservedRunningTime="2026-09-17 21:20:37.644314091 +0000 UTC m=+1902.796758467"
Sep 17 21:20:38.440343 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:38.440303    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997hwqk4"
Sep 17 21:20:46.440644 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:46.440610    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997hwqk4"
Sep 17 21:20:46.453235 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:46.453213    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997hwqk4"
Sep 17 21:20:46.679792 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:46.679711    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997hwqk4"
Sep 17 21:20:56.675576 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:56.675528    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:20:56.756286 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:56.756248    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:20:58.121453 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:58.121420    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997hwqk4"]
Sep 17 21:20:58.130578 ip-10-0-139-36 kubenswrapper[2575]: E0917 21:20:58.130544    2575 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 21:20:58.130578 ip-10-0-139-36 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:20:58.130578 ip-10-0-139-36 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 17 21:20:58.130578 ip-10-0-139-36 kubenswrapper[2575]:  > containerID="d92862d3f66c9d11f20bf8ec7a7a71213d9aaa76b5e9492d773e399d2aa0aae1" cmd=["/bin/sleep","15"]
Sep 17 21:20:58.130899 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:58.130596    2575 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997hwqk4" message="" err=<
Sep 17 21:20:58.130899 ip-10-0-139-36 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:20:58.130899 ip-10-0-139-36 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 17 21:20:58.130899 ip-10-0-139-36 kubenswrapper[2575]:  >
Sep 17 21:20:58.130899 ip-10-0-139-36 kubenswrapper[2575]: E0917 21:20:58.130608    2575 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 21:20:58.130899 ip-10-0-139-36 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:20:58.130899 ip-10-0-139-36 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 17 21:20:58.130899 ip-10-0-139-36 kubenswrapper[2575]:  > pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997hwqk4" podUID="e71f1d93-b032-41f9-bd45-3468ee381037" containerName="main" containerID="cri-o://d92862d3f66c9d11f20bf8ec7a7a71213d9aaa76b5e9492d773e399d2aa0aae1"
Sep 17 21:20:58.130899 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:58.130651    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997hwqk4" podUID="e71f1d93-b032-41f9-bd45-3468ee381037" containerName="main" containerID="cri-o://d92862d3f66c9d11f20bf8ec7a7a71213d9aaa76b5e9492d773e399d2aa0aae1" gracePeriod=60
Sep 17 21:20:58.170907 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:58.170873    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:20:58.485790 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:58.485767    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997hwqk4"
Sep 17 21:20:58.571527 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:58.571497    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-57ztx\" (UniqueName: \"kubernetes.io/projected/e71f1d93-b032-41f9-bd45-3468ee381037-kube-api-access-57ztx\") pod \"e71f1d93-b032-41f9-bd45-3468ee381037\" (UID: \"e71f1d93-b032-41f9-bd45-3468ee381037\") "
Sep 17 21:20:58.571726 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:58.571533    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e71f1d93-b032-41f9-bd45-3468ee381037-tls-certs\") pod \"e71f1d93-b032-41f9-bd45-3468ee381037\" (UID: \"e71f1d93-b032-41f9-bd45-3468ee381037\") "
Sep 17 21:20:58.571726 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:58.571610    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e71f1d93-b032-41f9-bd45-3468ee381037-home\") pod \"e71f1d93-b032-41f9-bd45-3468ee381037\" (UID: \"e71f1d93-b032-41f9-bd45-3468ee381037\") "
Sep 17 21:20:58.571726 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:58.571646    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e71f1d93-b032-41f9-bd45-3468ee381037-tmp-dir\") pod \"e71f1d93-b032-41f9-bd45-3468ee381037\" (UID: \"e71f1d93-b032-41f9-bd45-3468ee381037\") "
Sep 17 21:20:58.571726 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:58.571673    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e71f1d93-b032-41f9-bd45-3468ee381037-model-cache\") pod \"e71f1d93-b032-41f9-bd45-3468ee381037\" (UID: \"e71f1d93-b032-41f9-bd45-3468ee381037\") "
Sep 17 21:20:58.572030 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:58.571750    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e71f1d93-b032-41f9-bd45-3468ee381037-dshm\") pod \"e71f1d93-b032-41f9-bd45-3468ee381037\" (UID: \"e71f1d93-b032-41f9-bd45-3468ee381037\") "
Sep 17 21:20:58.572030 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:58.571903    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e71f1d93-b032-41f9-bd45-3468ee381037-home" (OuterVolumeSpecName: "home") pod "e71f1d93-b032-41f9-bd45-3468ee381037" (UID: "e71f1d93-b032-41f9-bd45-3468ee381037"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:20:58.572030 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:58.571918    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e71f1d93-b032-41f9-bd45-3468ee381037-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e71f1d93-b032-41f9-bd45-3468ee381037" (UID: "e71f1d93-b032-41f9-bd45-3468ee381037"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:20:58.572030 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:58.571957    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e71f1d93-b032-41f9-bd45-3468ee381037-model-cache" (OuterVolumeSpecName: "model-cache") pod "e71f1d93-b032-41f9-bd45-3468ee381037" (UID: "e71f1d93-b032-41f9-bd45-3468ee381037"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:20:58.572270 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:58.572071    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e71f1d93-b032-41f9-bd45-3468ee381037-tmp-dir\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:20:58.572270 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:58.572084    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e71f1d93-b032-41f9-bd45-3468ee381037-model-cache\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:20:58.572270 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:58.572095    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e71f1d93-b032-41f9-bd45-3468ee381037-home\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:20:58.574027 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:58.574004    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e71f1d93-b032-41f9-bd45-3468ee381037-dshm" (OuterVolumeSpecName: "dshm") pod "e71f1d93-b032-41f9-bd45-3468ee381037" (UID: "e71f1d93-b032-41f9-bd45-3468ee381037"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:20:58.574240 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:58.574215    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e71f1d93-b032-41f9-bd45-3468ee381037-kube-api-access-57ztx" (OuterVolumeSpecName: "kube-api-access-57ztx") pod "e71f1d93-b032-41f9-bd45-3468ee381037" (UID: "e71f1d93-b032-41f9-bd45-3468ee381037"). InnerVolumeSpecName "kube-api-access-57ztx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:20:58.574327 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:58.574308    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e71f1d93-b032-41f9-bd45-3468ee381037-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e71f1d93-b032-41f9-bd45-3468ee381037" (UID: "e71f1d93-b032-41f9-bd45-3468ee381037"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:20:58.673364 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:58.673281    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e71f1d93-b032-41f9-bd45-3468ee381037-dshm\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:20:58.673364 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:58.673316    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57ztx\" (UniqueName: \"kubernetes.io/projected/e71f1d93-b032-41f9-bd45-3468ee381037-kube-api-access-57ztx\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:20:58.673364 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:58.673327    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e71f1d93-b032-41f9-bd45-3468ee381037-tls-certs\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:20:58.729663 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:58.729635    2575 generic.go:356] "Generic (PLEG): container finished" podID="e71f1d93-b032-41f9-bd45-3468ee381037" containerID="d92862d3f66c9d11f20bf8ec7a7a71213d9aaa76b5e9492d773e399d2aa0aae1" exitCode=0
Sep 17 21:20:58.729814 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:58.729672    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997hwqk4" event={"ID":"e71f1d93-b032-41f9-bd45-3468ee381037","Type":"ContainerDied","Data":"d92862d3f66c9d11f20bf8ec7a7a71213d9aaa76b5e9492d773e399d2aa0aae1"}
Sep 17 21:20:58.729814 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:58.729683    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997hwqk4"
Sep 17 21:20:58.729814 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:58.729712    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997hwqk4" event={"ID":"e71f1d93-b032-41f9-bd45-3468ee381037","Type":"ContainerDied","Data":"e10a2a8b35a9b15826956c7eacbbe83f151440b2f1fcd81cbff3dd4c156d0336"}
Sep 17 21:20:58.729814 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:58.729733    2575 scope.go:117] "RemoveContainer" containerID="d92862d3f66c9d11f20bf8ec7a7a71213d9aaa76b5e9492d773e399d2aa0aae1"
Sep 17 21:20:58.746246 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:58.746225    2575 scope.go:117] "RemoveContainer" containerID="d92862d3f66c9d11f20bf8ec7a7a71213d9aaa76b5e9492d773e399d2aa0aae1"
Sep 17 21:20:58.746516 ip-10-0-139-36 kubenswrapper[2575]: E0917 21:20:58.746499    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d92862d3f66c9d11f20bf8ec7a7a71213d9aaa76b5e9492d773e399d2aa0aae1\": container with ID starting with d92862d3f66c9d11f20bf8ec7a7a71213d9aaa76b5e9492d773e399d2aa0aae1 not found: ID does not exist" containerID="d92862d3f66c9d11f20bf8ec7a7a71213d9aaa76b5e9492d773e399d2aa0aae1"
Sep 17 21:20:58.746564 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:58.746523    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d92862d3f66c9d11f20bf8ec7a7a71213d9aaa76b5e9492d773e399d2aa0aae1"} err="failed to get container status \"d92862d3f66c9d11f20bf8ec7a7a71213d9aaa76b5e9492d773e399d2aa0aae1\": rpc error: code = NotFound desc = could not find container \"d92862d3f66c9d11f20bf8ec7a7a71213d9aaa76b5e9492d773e399d2aa0aae1\": container with ID starting with d92862d3f66c9d11f20bf8ec7a7a71213d9aaa76b5e9492d773e399d2aa0aae1 not found: ID does not exist"
Sep 17 21:20:58.782529 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:58.782503    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997hwqk4"]
Sep 17 21:20:58.803897 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:58.803871    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997hwqk4"]
Sep 17 21:20:59.357027 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:20:59.356998    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e71f1d93-b032-41f9-bd45-3468ee381037" path="/var/lib/kubelet/pods/e71f1d93-b032-41f9-bd45-3468ee381037/volumes"
Sep 17 21:21:02.701337 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:21:02.701304    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:21:02.801627 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:21:02.801589    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:21:07.741041 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:21:07.741001    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:21:21.353143 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:21:21.353104    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:21:21.694990 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:21:21.694898    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:21:28.396494 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:21:28.396460    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:21:28.801006 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:21:28.800966    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:21:30.357046 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:21:30.356946    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:21:36.355453 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:21:36.355421    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:21:44.538539 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:21:44.538507    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:21:44.744813 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:21:44.744775    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:21:50.197189 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:21:50.197154    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:21:51.742266 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:21:51.742227    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:21:57.476942 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:21:57.476905    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:21:57.627004 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:21:57.626974    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:22:17.488185 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:22:17.488156    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v9mr2"]
Sep 17 21:22:17.488692 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:22:17.488540    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e71f1d93-b032-41f9-bd45-3468ee381037" containerName="main"
Sep 17 21:22:17.488692 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:22:17.488553    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e71f1d93-b032-41f9-bd45-3468ee381037" containerName="main"
Sep 17 21:22:17.488692 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:22:17.488623    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e71f1d93-b032-41f9-bd45-3468ee381037" containerName="main"
Sep 17 21:22:17.491673 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:22:17.491654    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v9mr2"
Sep 17 21:22:17.495515 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:22:17.495493    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Sep 17 21:22:17.495654 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:22:17.495493    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Sep 17 21:22:17.495654 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:22:17.495543    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-6mjf7\""
Sep 17 21:22:17.495654 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:22:17.495493    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Sep 17 21:22:17.505362 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:22:17.505338    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v9mr2"]
Sep 17 21:22:17.565594 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:22:17.565564    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plwfg\" (UniqueName: \"kubernetes.io/projected/901f1e94-a889-4819-a951-25837260cb1e-kube-api-access-plwfg\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v9mr2\" (UID: \"901f1e94-a889-4819-a951-25837260cb1e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v9mr2"
Sep 17 21:22:17.565594 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:22:17.565599    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/901f1e94-a889-4819-a951-25837260cb1e-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v9mr2\" (UID: \"901f1e94-a889-4819-a951-25837260cb1e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v9mr2"
Sep 17 21:22:17.565901 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:22:17.565622    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/901f1e94-a889-4819-a951-25837260cb1e-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v9mr2\" (UID: \"901f1e94-a889-4819-a951-25837260cb1e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v9mr2"
Sep 17 21:22:17.565901 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:22:17.565682    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/901f1e94-a889-4819-a951-25837260cb1e-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v9mr2\" (UID: \"901f1e94-a889-4819-a951-25837260cb1e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v9mr2"
Sep 17 21:22:17.565901 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:22:17.565715    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/901f1e94-a889-4819-a951-25837260cb1e-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v9mr2\" (UID: \"901f1e94-a889-4819-a951-25837260cb1e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v9mr2"
Sep 17 21:22:17.565901 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:22:17.565774    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/901f1e94-a889-4819-a951-25837260cb1e-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v9mr2\" (UID: \"901f1e94-a889-4819-a951-25837260cb1e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v9mr2"
Sep 17 21:22:17.576050 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:22:17.576015    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:22:17.666741 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:22:17.666703    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/901f1e94-a889-4819-a951-25837260cb1e-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v9mr2\" (UID: \"901f1e94-a889-4819-a951-25837260cb1e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v9mr2"
Sep 17 21:22:17.666935 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:22:17.666756    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plwfg\" (UniqueName: \"kubernetes.io/projected/901f1e94-a889-4819-a951-25837260cb1e-kube-api-access-plwfg\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v9mr2\" (UID: \"901f1e94-a889-4819-a951-25837260cb1e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v9mr2"
Sep 17 21:22:17.666935 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:22:17.666779    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/901f1e94-a889-4819-a951-25837260cb1e-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v9mr2\" (UID: \"901f1e94-a889-4819-a951-25837260cb1e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v9mr2"
Sep 17 21:22:17.666935 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:22:17.666797    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/901f1e94-a889-4819-a951-25837260cb1e-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v9mr2\" (UID: \"901f1e94-a889-4819-a951-25837260cb1e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v9mr2"
Sep 17 21:22:17.666935 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:22:17.666841    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/901f1e94-a889-4819-a951-25837260cb1e-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v9mr2\" (UID: \"901f1e94-a889-4819-a951-25837260cb1e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v9mr2"
Sep 17 21:22:17.666935 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:22:17.666906    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/901f1e94-a889-4819-a951-25837260cb1e-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v9mr2\" (UID: \"901f1e94-a889-4819-a951-25837260cb1e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v9mr2"
Sep 17 21:22:17.667244 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:22:17.667216    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/901f1e94-a889-4819-a951-25837260cb1e-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v9mr2\" (UID: \"901f1e94-a889-4819-a951-25837260cb1e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v9mr2"
Sep 17 21:22:17.667306 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:22:17.667265    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/901f1e94-a889-4819-a951-25837260cb1e-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v9mr2\" (UID: \"901f1e94-a889-4819-a951-25837260cb1e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v9mr2"
Sep 17 21:22:17.667363 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:22:17.667343    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/901f1e94-a889-4819-a951-25837260cb1e-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v9mr2\" (UID: \"901f1e94-a889-4819-a951-25837260cb1e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v9mr2"
Sep 17 21:22:17.669194 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:22:17.669171    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/901f1e94-a889-4819-a951-25837260cb1e-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v9mr2\" (UID: \"901f1e94-a889-4819-a951-25837260cb1e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v9mr2"
Sep 17 21:22:17.669490 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:22:17.669471    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/901f1e94-a889-4819-a951-25837260cb1e-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v9mr2\" (UID: \"901f1e94-a889-4819-a951-25837260cb1e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v9mr2"
Sep 17 21:22:17.678417 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:22:17.678392    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plwfg\" (UniqueName: \"kubernetes.io/projected/901f1e94-a889-4819-a951-25837260cb1e-kube-api-access-plwfg\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v9mr2\" (UID: \"901f1e94-a889-4819-a951-25837260cb1e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v9mr2"
Sep 17 21:22:17.804755 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:22:17.804666    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v9mr2"
Sep 17 21:22:17.927933 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:22:17.927897    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:22:18.086198 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:22:18.086172    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v9mr2"]
Sep 17 21:22:18.088872 ip-10-0-139-36 kubenswrapper[2575]: W0917 21:22:18.088819    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod901f1e94_a889_4819_a951_25837260cb1e.slice/crio-f57a46ce1766543cbc5838dcb28c5a9d4741ae868e3a7e37018d1a28c1052e28 WatchSource:0}: Error finding container f57a46ce1766543cbc5838dcb28c5a9d4741ae868e3a7e37018d1a28c1052e28: Status 404 returned error can't find the container with id f57a46ce1766543cbc5838dcb28c5a9d4741ae868e3a7e37018d1a28c1052e28
Sep 17 21:22:18.115380 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:22:18.115352    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v9mr2" event={"ID":"901f1e94-a889-4819-a951-25837260cb1e","Type":"ContainerStarted","Data":"f57a46ce1766543cbc5838dcb28c5a9d4741ae868e3a7e37018d1a28c1052e28"}
Sep 17 21:22:19.122290 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:22:19.122254    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v9mr2" event={"ID":"901f1e94-a889-4819-a951-25837260cb1e","Type":"ContainerStarted","Data":"5dc1cb1a7c220903c51415cfc13d7460c0e4f447e65d34c614728c54d85cbcc8"}
Sep 17 21:22:19.140983 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:22:19.140931    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v9mr2" podStartSLOduration=2.140914558 podStartE2EDuration="2.140914558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:22:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:22:19.139066359 +0000 UTC m=+2004.291510735" watchObservedRunningTime="2026-09-17 21:22:19.140914558 +0000 UTC m=+2004.293358934"
Sep 17 21:22:19.804947 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:22:19.804898    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v9mr2"
Sep 17 21:22:27.805403 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:22:27.805366    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v9mr2"
Sep 17 21:22:27.817825 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:22:27.817803    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v9mr2"
Sep 17 21:22:28.177785 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:22:28.177704    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v9mr2"
Sep 17 21:22:28.672104 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:22:28.672067    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:23:11.835499 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:23:11.835464    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:23:12.033220 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:23:12.033181    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:23:21.748749 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:23:21.748661    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:23:45.260533 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:23:45.260495    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:23:45.317485 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:23:45.317456    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:23:48.017299 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:23:48.017245    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:23:48.114384 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:23:48.114347    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:23:50.435521 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:23:50.435476    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:23:50.536341 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:23:50.536306    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:24:35.231240 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:24:35.231210    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v9mr2"]
Sep 17 21:24:35.234133 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:24:35.234097    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:24:35.240965 ip-10-0-139-36 kubenswrapper[2575]: E0917 21:24:35.240932    2575 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 21:24:35.240965 ip-10-0-139-36 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:24:35.240965 ip-10-0-139-36 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 17 21:24:35.240965 ip-10-0-139-36 kubenswrapper[2575]:  > containerID="5dc1cb1a7c220903c51415cfc13d7460c0e4f447e65d34c614728c54d85cbcc8" cmd=["/bin/sleep","15"]
Sep 17 21:24:35.241190 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:24:35.240992    2575 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v9mr2" message="" err=<
Sep 17 21:24:35.241190 ip-10-0-139-36 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:24:35.241190 ip-10-0-139-36 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 17 21:24:35.241190 ip-10-0-139-36 kubenswrapper[2575]:  >
Sep 17 21:24:35.241190 ip-10-0-139-36 kubenswrapper[2575]: E0917 21:24:35.241008    2575 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 21:24:35.241190 ip-10-0-139-36 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:24:35.241190 ip-10-0-139-36 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 17 21:24:35.241190 ip-10-0-139-36 kubenswrapper[2575]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v9mr2" podUID="901f1e94-a889-4819-a951-25837260cb1e" containerName="main" containerID="cri-o://5dc1cb1a7c220903c51415cfc13d7460c0e4f447e65d34c614728c54d85cbcc8"
Sep 17 21:24:35.241190 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:24:35.241047    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v9mr2" podUID="901f1e94-a889-4819-a951-25837260cb1e" containerName="main" containerID="cri-o://5dc1cb1a7c220903c51415cfc13d7460c0e4f447e65d34c614728c54d85cbcc8" gracePeriod=60
Sep 17 21:24:35.327547 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:24:35.327515    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:24:35.805339 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:24:35.805312    2575 generic.go:356] "Generic (PLEG): container finished" podID="901f1e94-a889-4819-a951-25837260cb1e" containerID="5dc1cb1a7c220903c51415cfc13d7460c0e4f447e65d34c614728c54d85cbcc8" exitCode=0
Sep 17 21:24:35.805509 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:24:35.805395    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v9mr2" event={"ID":"901f1e94-a889-4819-a951-25837260cb1e","Type":"ContainerDied","Data":"5dc1cb1a7c220903c51415cfc13d7460c0e4f447e65d34c614728c54d85cbcc8"}
Sep 17 21:24:35.983094 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:24:35.983070    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v9mr2"
Sep 17 21:24:36.001089 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:24:36.001063    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/901f1e94-a889-4819-a951-25837260cb1e-tls-certs\") pod \"901f1e94-a889-4819-a951-25837260cb1e\" (UID: \"901f1e94-a889-4819-a951-25837260cb1e\") "
Sep 17 21:24:36.001321 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:24:36.001304    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/901f1e94-a889-4819-a951-25837260cb1e-tmp-dir\") pod \"901f1e94-a889-4819-a951-25837260cb1e\" (UID: \"901f1e94-a889-4819-a951-25837260cb1e\") "
Sep 17 21:24:36.001450 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:24:36.001430    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/901f1e94-a889-4819-a951-25837260cb1e-model-cache\") pod \"901f1e94-a889-4819-a951-25837260cb1e\" (UID: \"901f1e94-a889-4819-a951-25837260cb1e\") "
Sep 17 21:24:36.001570 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:24:36.001489    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/901f1e94-a889-4819-a951-25837260cb1e-dshm\") pod \"901f1e94-a889-4819-a951-25837260cb1e\" (UID: \"901f1e94-a889-4819-a951-25837260cb1e\") "
Sep 17 21:24:36.001570 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:24:36.001534    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/901f1e94-a889-4819-a951-25837260cb1e-home\") pod \"901f1e94-a889-4819-a951-25837260cb1e\" (UID: \"901f1e94-a889-4819-a951-25837260cb1e\") "
Sep 17 21:24:36.001687 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:24:36.001562    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/901f1e94-a889-4819-a951-25837260cb1e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "901f1e94-a889-4819-a951-25837260cb1e" (UID: "901f1e94-a889-4819-a951-25837260cb1e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:24:36.001687 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:24:36.001569    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-plwfg\" (UniqueName: \"kubernetes.io/projected/901f1e94-a889-4819-a951-25837260cb1e-kube-api-access-plwfg\") pod \"901f1e94-a889-4819-a951-25837260cb1e\" (UID: \"901f1e94-a889-4819-a951-25837260cb1e\") "
Sep 17 21:24:36.001800 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:24:36.001741    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/901f1e94-a889-4819-a951-25837260cb1e-model-cache" (OuterVolumeSpecName: "model-cache") pod "901f1e94-a889-4819-a951-25837260cb1e" (UID: "901f1e94-a889-4819-a951-25837260cb1e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:24:36.001997 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:24:36.001964    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/901f1e94-a889-4819-a951-25837260cb1e-home" (OuterVolumeSpecName: "home") pod "901f1e94-a889-4819-a951-25837260cb1e" (UID: "901f1e94-a889-4819-a951-25837260cb1e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:24:36.002093 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:24:36.001995    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/901f1e94-a889-4819-a951-25837260cb1e-tmp-dir\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:24:36.002093 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:24:36.002018    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/901f1e94-a889-4819-a951-25837260cb1e-model-cache\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:24:36.003931 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:24:36.003840    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/901f1e94-a889-4819-a951-25837260cb1e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "901f1e94-a889-4819-a951-25837260cb1e" (UID: "901f1e94-a889-4819-a951-25837260cb1e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:24:36.004165 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:24:36.004133    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/901f1e94-a889-4819-a951-25837260cb1e-kube-api-access-plwfg" (OuterVolumeSpecName: "kube-api-access-plwfg") pod "901f1e94-a889-4819-a951-25837260cb1e" (UID: "901f1e94-a889-4819-a951-25837260cb1e"). InnerVolumeSpecName "kube-api-access-plwfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:24:36.004253 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:24:36.004161    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/901f1e94-a889-4819-a951-25837260cb1e-dshm" (OuterVolumeSpecName: "dshm") pod "901f1e94-a889-4819-a951-25837260cb1e" (UID: "901f1e94-a889-4819-a951-25837260cb1e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:24:36.102462 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:24:36.102390    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/901f1e94-a889-4819-a951-25837260cb1e-tls-certs\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:24:36.102462 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:24:36.102416    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/901f1e94-a889-4819-a951-25837260cb1e-dshm\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:24:36.102462 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:24:36.102426    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/901f1e94-a889-4819-a951-25837260cb1e-home\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:24:36.102462 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:24:36.102434    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plwfg\" (UniqueName: \"kubernetes.io/projected/901f1e94-a889-4819-a951-25837260cb1e-kube-api-access-plwfg\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:24:36.566414 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:24:36.566372    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:24:36.662168 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:24:36.662131    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:24:36.811183 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:24:36.811154    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v9mr2"
Sep 17 21:24:36.811367 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:24:36.811203    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v9mr2" event={"ID":"901f1e94-a889-4819-a951-25837260cb1e","Type":"ContainerDied","Data":"f57a46ce1766543cbc5838dcb28c5a9d4741ae868e3a7e37018d1a28c1052e28"}
Sep 17 21:24:36.811367 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:24:36.811261    2575 scope.go:117] "RemoveContainer" containerID="5dc1cb1a7c220903c51415cfc13d7460c0e4f447e65d34c614728c54d85cbcc8"
Sep 17 21:24:36.836727 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:24:36.836693    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v9mr2"]
Sep 17 21:24:36.840388 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:24:36.840366    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v9mr2"]
Sep 17 21:24:37.357314 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:24:37.357278    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="901f1e94-a889-4819-a951-25837260cb1e" path="/var/lib/kubelet/pods/901f1e94-a889-4819-a951-25837260cb1e/volumes"
Sep 17 21:24:43.441947 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:24:43.441911    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:24:48.527784 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:24:48.527700    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:24:48.742114 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:24:48.742075    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:24:50.022260 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:24:50.022221    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:24:50.118229 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:24:50.118199    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:24:54.910228 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:24:54.910186    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:25:10.477744 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:25:10.477694    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:25:25.265327 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:25:25.265290    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:25:25.365291 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:25:25.365257    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:25:27.889509 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:25:27.889468    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:25:27.991291 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:25:27.991250    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:25:31.807295 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:25:31.807244    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796967nn2"]
Sep 17 21:25:31.807743 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:25:31.807663    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="901f1e94-a889-4819-a951-25837260cb1e" containerName="main"
Sep 17 21:25:31.807743 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:25:31.807680    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="901f1e94-a889-4819-a951-25837260cb1e" containerName="main"
Sep 17 21:25:31.807820 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:25:31.807749    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="901f1e94-a889-4819-a951-25837260cb1e" containerName="main"
Sep 17 21:25:31.810726 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:25:31.810701    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796967nn2"
Sep 17 21:25:31.813245 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:25:31.813214    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-5pmv8\""
Sep 17 21:25:31.813366 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:25:31.813217    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 17 21:25:31.815382 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:25:31.815358    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 17 21:25:31.815550 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:25:31.815426    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Sep 17 21:25:31.820337 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:25:31.820316    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796967nn2"]
Sep 17 21:25:31.891404 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:25:31.891373    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69e49267-cd70-4617-bfa2-f4bc95ed2598-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796967nn2\" (UID: \"69e49267-cd70-4617-bfa2-f4bc95ed2598\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796967nn2"
Sep 17 21:25:31.891404 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:25:31.891412    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69e49267-cd70-4617-bfa2-f4bc95ed2598-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796967nn2\" (UID: \"69e49267-cd70-4617-bfa2-f4bc95ed2598\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796967nn2"
Sep 17 21:25:31.891616 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:25:31.891475    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69e49267-cd70-4617-bfa2-f4bc95ed2598-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796967nn2\" (UID: \"69e49267-cd70-4617-bfa2-f4bc95ed2598\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796967nn2"
Sep 17 21:25:31.891616 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:25:31.891550    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njbnn\" (UniqueName: \"kubernetes.io/projected/69e49267-cd70-4617-bfa2-f4bc95ed2598-kube-api-access-njbnn\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796967nn2\" (UID: \"69e49267-cd70-4617-bfa2-f4bc95ed2598\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796967nn2"
Sep 17 21:25:31.891690 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:25:31.891618    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69e49267-cd70-4617-bfa2-f4bc95ed2598-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796967nn2\" (UID: \"69e49267-cd70-4617-bfa2-f4bc95ed2598\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796967nn2"
Sep 17 21:25:31.891690 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:25:31.891662    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69e49267-cd70-4617-bfa2-f4bc95ed2598-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796967nn2\" (UID: \"69e49267-cd70-4617-bfa2-f4bc95ed2598\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796967nn2"
Sep 17 21:25:31.993064 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:25:31.993031    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69e49267-cd70-4617-bfa2-f4bc95ed2598-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796967nn2\" (UID: \"69e49267-cd70-4617-bfa2-f4bc95ed2598\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796967nn2"
Sep 17 21:25:31.993251 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:25:31.993072    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69e49267-cd70-4617-bfa2-f4bc95ed2598-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796967nn2\" (UID: \"69e49267-cd70-4617-bfa2-f4bc95ed2598\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796967nn2"
Sep 17 21:25:31.993251 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:25:31.993102    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69e49267-cd70-4617-bfa2-f4bc95ed2598-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796967nn2\" (UID: \"69e49267-cd70-4617-bfa2-f4bc95ed2598\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796967nn2"
Sep 17 21:25:31.993251 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:25:31.993134    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njbnn\" (UniqueName: \"kubernetes.io/projected/69e49267-cd70-4617-bfa2-f4bc95ed2598-kube-api-access-njbnn\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796967nn2\" (UID: \"69e49267-cd70-4617-bfa2-f4bc95ed2598\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796967nn2"
Sep 17 21:25:31.993251 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:25:31.993168    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69e49267-cd70-4617-bfa2-f4bc95ed2598-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796967nn2\" (UID: \"69e49267-cd70-4617-bfa2-f4bc95ed2598\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796967nn2"
Sep 17 21:25:31.993251 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:25:31.993204    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69e49267-cd70-4617-bfa2-f4bc95ed2598-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796967nn2\" (UID: \"69e49267-cd70-4617-bfa2-f4bc95ed2598\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796967nn2"
Sep 17 21:25:31.993544 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:25:31.993490    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69e49267-cd70-4617-bfa2-f4bc95ed2598-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796967nn2\" (UID: \"69e49267-cd70-4617-bfa2-f4bc95ed2598\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796967nn2"
Sep 17 21:25:31.993544 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:25:31.993529    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69e49267-cd70-4617-bfa2-f4bc95ed2598-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796967nn2\" (UID: \"69e49267-cd70-4617-bfa2-f4bc95ed2598\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796967nn2"
Sep 17 21:25:31.993624 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:25:31.993570    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69e49267-cd70-4617-bfa2-f4bc95ed2598-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796967nn2\" (UID: \"69e49267-cd70-4617-bfa2-f4bc95ed2598\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796967nn2"
Sep 17 21:25:31.995399 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:25:31.995379    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69e49267-cd70-4617-bfa2-f4bc95ed2598-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796967nn2\" (UID: \"69e49267-cd70-4617-bfa2-f4bc95ed2598\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796967nn2"
Sep 17 21:25:31.995811 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:25:31.995788    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69e49267-cd70-4617-bfa2-f4bc95ed2598-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796967nn2\" (UID: \"69e49267-cd70-4617-bfa2-f4bc95ed2598\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796967nn2"
Sep 17 21:25:31.997157 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:25:31.997125    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:25:32.001653 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:25:32.001631    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njbnn\" (UniqueName: \"kubernetes.io/projected/69e49267-cd70-4617-bfa2-f4bc95ed2598-kube-api-access-njbnn\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796967nn2\" (UID: \"69e49267-cd70-4617-bfa2-f4bc95ed2598\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796967nn2"
Sep 17 21:25:32.122266 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:25:32.122175    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796967nn2"
Sep 17 21:25:32.359963 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:25:32.359926    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:25:32.395313 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:25:32.395221    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796967nn2"]
Sep 17 21:25:32.398755 ip-10-0-139-36 kubenswrapper[2575]: W0917 21:25:32.398731    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69e49267_cd70_4617_bfa2_f4bc95ed2598.slice/crio-363492a8e043f0c629ad755702aa4c36aa20a0c6750752d7e4f2d3dd87259fa4 WatchSource:0}: Error finding container 363492a8e043f0c629ad755702aa4c36aa20a0c6750752d7e4f2d3dd87259fa4: Status 404 returned error can't find the container with id 363492a8e043f0c629ad755702aa4c36aa20a0c6750752d7e4f2d3dd87259fa4
Sep 17 21:25:32.635952 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:25:32.635912    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:25:33.089872 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:25:33.089809    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796967nn2" event={"ID":"69e49267-cd70-4617-bfa2-f4bc95ed2598","Type":"ContainerStarted","Data":"0004657109a1b0e9d9b58f0e5e6eb437723820348741d3165710c82c8d5f3077"}
Sep 17 21:25:33.090353 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:25:33.089903    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796967nn2" event={"ID":"69e49267-cd70-4617-bfa2-f4bc95ed2598","Type":"ContainerStarted","Data":"363492a8e043f0c629ad755702aa4c36aa20a0c6750752d7e4f2d3dd87259fa4"}
Sep 17 21:25:33.121272 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:25:33.121207    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796967nn2" podStartSLOduration=2.121184795 podStartE2EDuration="2.121184795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:25:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:25:33.118020574 +0000 UTC m=+2198.270464951" watchObservedRunningTime="2026-09-17 21:25:33.121184795 +0000 UTC m=+2198.273629173"
Sep 17 21:25:33.123014 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:25:33.122982    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796967nn2"
Sep 17 21:25:38.618340 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:25:38.618286    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:25:38.719275 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:25:38.719240    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:25:42.122980 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:25:42.122942    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796967nn2"
Sep 17 21:25:42.135786 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:25:42.135760    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796967nn2"
Sep 17 21:25:42.147020 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:25:42.147001    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796967nn2"
Sep 17 21:25:44.322667 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:25:44.322604    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:25:47.076384 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:25:47.076347    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:25:47.161341 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:25:47.161306    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:26:19.794974 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:26:19.794894    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sk228"]
Sep 17 21:26:19.798529 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:26:19.798510    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sk228"
Sep 17 21:26:19.800983 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:26:19.800959    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Sep 17 21:26:19.809606 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:26:19.809582    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sk228"]
Sep 17 21:26:19.822627 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:26:19.822602    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:26:19.944665 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:26:19.944632    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/549f7592-f9b1-45a9-a1af-3bfd3964e3d9-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sk228\" (UID: \"549f7592-f9b1-45a9-a1af-3bfd3964e3d9\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sk228"
Sep 17 21:26:19.944894 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:26:19.944680    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/549f7592-f9b1-45a9-a1af-3bfd3964e3d9-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sk228\" (UID: \"549f7592-f9b1-45a9-a1af-3bfd3964e3d9\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sk228"
Sep 17 21:26:19.944894 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:26:19.944700    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/549f7592-f9b1-45a9-a1af-3bfd3964e3d9-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sk228\" (UID: \"549f7592-f9b1-45a9-a1af-3bfd3964e3d9\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sk228"
Sep 17 21:26:19.944894 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:26:19.944766    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/549f7592-f9b1-45a9-a1af-3bfd3964e3d9-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sk228\" (UID: \"549f7592-f9b1-45a9-a1af-3bfd3964e3d9\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sk228"
Sep 17 21:26:19.944894 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:26:19.944781    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/549f7592-f9b1-45a9-a1af-3bfd3964e3d9-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sk228\" (UID: \"549f7592-f9b1-45a9-a1af-3bfd3964e3d9\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sk228"
Sep 17 21:26:19.944894 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:26:19.944804    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbg4x\" (UniqueName: \"kubernetes.io/projected/549f7592-f9b1-45a9-a1af-3bfd3964e3d9-kube-api-access-sbg4x\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sk228\" (UID: \"549f7592-f9b1-45a9-a1af-3bfd3964e3d9\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sk228"
Sep 17 21:26:20.046076 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:26:20.045982    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbg4x\" (UniqueName: \"kubernetes.io/projected/549f7592-f9b1-45a9-a1af-3bfd3964e3d9-kube-api-access-sbg4x\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sk228\" (UID: \"549f7592-f9b1-45a9-a1af-3bfd3964e3d9\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sk228"
Sep 17 21:26:20.046076 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:26:20.046056    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/549f7592-f9b1-45a9-a1af-3bfd3964e3d9-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sk228\" (UID: \"549f7592-f9b1-45a9-a1af-3bfd3964e3d9\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sk228"
Sep 17 21:26:20.046307 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:26:20.046087    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/549f7592-f9b1-45a9-a1af-3bfd3964e3d9-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sk228\" (UID: \"549f7592-f9b1-45a9-a1af-3bfd3964e3d9\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sk228"
Sep 17 21:26:20.046307 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:26:20.046106    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/549f7592-f9b1-45a9-a1af-3bfd3964e3d9-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sk228\" (UID: \"549f7592-f9b1-45a9-a1af-3bfd3964e3d9\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sk228"
Sep 17 21:26:20.046307 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:26:20.046145    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/549f7592-f9b1-45a9-a1af-3bfd3964e3d9-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sk228\" (UID: \"549f7592-f9b1-45a9-a1af-3bfd3964e3d9\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sk228"
Sep 17 21:26:20.046307 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:26:20.046169    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/549f7592-f9b1-45a9-a1af-3bfd3964e3d9-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sk228\" (UID: \"549f7592-f9b1-45a9-a1af-3bfd3964e3d9\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sk228"
Sep 17 21:26:20.046569 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:26:20.046546    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/549f7592-f9b1-45a9-a1af-3bfd3964e3d9-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sk228\" (UID: \"549f7592-f9b1-45a9-a1af-3bfd3964e3d9\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sk228"
Sep 17 21:26:20.046632 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:26:20.046583    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/549f7592-f9b1-45a9-a1af-3bfd3964e3d9-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sk228\" (UID: \"549f7592-f9b1-45a9-a1af-3bfd3964e3d9\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sk228"
Sep 17 21:26:20.046691 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:26:20.046626    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/549f7592-f9b1-45a9-a1af-3bfd3964e3d9-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sk228\" (UID: \"549f7592-f9b1-45a9-a1af-3bfd3964e3d9\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sk228"
Sep 17 21:26:20.048688 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:26:20.048662    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/549f7592-f9b1-45a9-a1af-3bfd3964e3d9-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sk228\" (UID: \"549f7592-f9b1-45a9-a1af-3bfd3964e3d9\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sk228"
Sep 17 21:26:20.048812 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:26:20.048798    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/549f7592-f9b1-45a9-a1af-3bfd3964e3d9-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sk228\" (UID: \"549f7592-f9b1-45a9-a1af-3bfd3964e3d9\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sk228"
Sep 17 21:26:20.059551 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:26:20.059528    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbg4x\" (UniqueName: \"kubernetes.io/projected/549f7592-f9b1-45a9-a1af-3bfd3964e3d9-kube-api-access-sbg4x\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sk228\" (UID: \"549f7592-f9b1-45a9-a1af-3bfd3964e3d9\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sk228"
Sep 17 21:26:20.087384 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:26:20.087348    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:26:20.110361 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:26:20.110335    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sk228"
Sep 17 21:26:20.386003 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:26:20.385975    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sk228"]
Sep 17 21:26:20.388207 ip-10-0-139-36 kubenswrapper[2575]: W0917 21:26:20.388179    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod549f7592_f9b1_45a9_a1af_3bfd3964e3d9.slice/crio-d7a74d3426498df96768d34c0ad0b370b0efdcbae8b0af1bcc6a1b7a0096d911 WatchSource:0}: Error finding container d7a74d3426498df96768d34c0ad0b370b0efdcbae8b0af1bcc6a1b7a0096d911: Status 404 returned error can't find the container with id d7a74d3426498df96768d34c0ad0b370b0efdcbae8b0af1bcc6a1b7a0096d911
Sep 17 21:26:21.326293 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:26:21.326243    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sk228" event={"ID":"549f7592-f9b1-45a9-a1af-3bfd3964e3d9","Type":"ContainerStarted","Data":"07660cf5542f9133c13ad1973fcd45228d6d7ba6f78e0711b9e616f50c516ecf"}
Sep 17 21:26:21.326809 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:26:21.326328    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sk228" event={"ID":"549f7592-f9b1-45a9-a1af-3bfd3964e3d9","Type":"ContainerStarted","Data":"d7a74d3426498df96768d34c0ad0b370b0efdcbae8b0af1bcc6a1b7a0096d911"}
Sep 17 21:26:21.350826 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:26:21.350753    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sk228" podStartSLOduration=2.350737862 podStartE2EDuration="2.350737862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:26:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:26:21.347818338 +0000 UTC m=+2246.500262739" watchObservedRunningTime="2026-09-17 21:26:21.350737862 +0000 UTC m=+2246.503182268"
Sep 17 21:26:22.110899 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:26:22.110843    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sk228"
Sep 17 21:26:29.240875 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:26:29.240831    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:26:30.110903 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:26:30.110843    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sk228"
Sep 17 21:26:30.123597 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:26:30.123565    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sk228"
Sep 17 21:26:30.383185 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:26:30.383097    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sk228"
Sep 17 21:26:54.251240 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:26:54.251201    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:26:54.628520 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:26:54.628430    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sk228"]
Sep 17 21:26:54.637394 ip-10-0-139-36 kubenswrapper[2575]: E0917 21:26:54.637361    2575 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 21:26:54.637394 ip-10-0-139-36 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:26:54.637394 ip-10-0-139-36 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 17 21:26:54.637394 ip-10-0-139-36 kubenswrapper[2575]:  > containerID="07660cf5542f9133c13ad1973fcd45228d6d7ba6f78e0711b9e616f50c516ecf" cmd=["/bin/sleep","15"]
Sep 17 21:26:54.637625 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:26:54.637411    2575 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sk228" message="" err=<
Sep 17 21:26:54.637625 ip-10-0-139-36 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:26:54.637625 ip-10-0-139-36 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 17 21:26:54.637625 ip-10-0-139-36 kubenswrapper[2575]:  >
Sep 17 21:26:54.637625 ip-10-0-139-36 kubenswrapper[2575]: E0917 21:26:54.637423    2575 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 21:26:54.637625 ip-10-0-139-36 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:26:54.637625 ip-10-0-139-36 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 17 21:26:54.637625 ip-10-0-139-36 kubenswrapper[2575]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sk228" podUID="549f7592-f9b1-45a9-a1af-3bfd3964e3d9" containerName="main" containerID="cri-o://07660cf5542f9133c13ad1973fcd45228d6d7ba6f78e0711b9e616f50c516ecf"
Sep 17 21:26:54.637625 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:26:54.637456    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sk228" podUID="549f7592-f9b1-45a9-a1af-3bfd3964e3d9" containerName="main" containerID="cri-o://07660cf5542f9133c13ad1973fcd45228d6d7ba6f78e0711b9e616f50c516ecf" gracePeriod=60
Sep 17 21:26:54.990561 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:26:54.990535    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sk228"
Sep 17 21:26:55.051319 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:26:55.051295    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sbg4x\" (UniqueName: \"kubernetes.io/projected/549f7592-f9b1-45a9-a1af-3bfd3964e3d9-kube-api-access-sbg4x\") pod \"549f7592-f9b1-45a9-a1af-3bfd3964e3d9\" (UID: \"549f7592-f9b1-45a9-a1af-3bfd3964e3d9\") "
Sep 17 21:26:55.051450 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:26:55.051383    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/549f7592-f9b1-45a9-a1af-3bfd3964e3d9-dshm\") pod \"549f7592-f9b1-45a9-a1af-3bfd3964e3d9\" (UID: \"549f7592-f9b1-45a9-a1af-3bfd3964e3d9\") "
Sep 17 21:26:55.051450 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:26:55.051417    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/549f7592-f9b1-45a9-a1af-3bfd3964e3d9-tmp-dir\") pod \"549f7592-f9b1-45a9-a1af-3bfd3964e3d9\" (UID: \"549f7592-f9b1-45a9-a1af-3bfd3964e3d9\") "
Sep 17 21:26:55.051524 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:26:55.051469    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/549f7592-f9b1-45a9-a1af-3bfd3964e3d9-tls-certs\") pod \"549f7592-f9b1-45a9-a1af-3bfd3964e3d9\" (UID: \"549f7592-f9b1-45a9-a1af-3bfd3964e3d9\") "
Sep 17 21:26:55.051524 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:26:55.051492    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/549f7592-f9b1-45a9-a1af-3bfd3964e3d9-model-cache\") pod \"549f7592-f9b1-45a9-a1af-3bfd3964e3d9\" (UID: \"549f7592-f9b1-45a9-a1af-3bfd3964e3d9\") "
Sep 17 21:26:55.051616 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:26:55.051528    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/549f7592-f9b1-45a9-a1af-3bfd3964e3d9-home\") pod \"549f7592-f9b1-45a9-a1af-3bfd3964e3d9\" (UID: \"549f7592-f9b1-45a9-a1af-3bfd3964e3d9\") "
Sep 17 21:26:55.051804 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:26:55.051767    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/549f7592-f9b1-45a9-a1af-3bfd3964e3d9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "549f7592-f9b1-45a9-a1af-3bfd3964e3d9" (UID: "549f7592-f9b1-45a9-a1af-3bfd3964e3d9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:26:55.051804 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:26:55.051782    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/549f7592-f9b1-45a9-a1af-3bfd3964e3d9-model-cache" (OuterVolumeSpecName: "model-cache") pod "549f7592-f9b1-45a9-a1af-3bfd3964e3d9" (UID: "549f7592-f9b1-45a9-a1af-3bfd3964e3d9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:26:55.051995 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:26:55.051945    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/549f7592-f9b1-45a9-a1af-3bfd3964e3d9-home" (OuterVolumeSpecName: "home") pod "549f7592-f9b1-45a9-a1af-3bfd3964e3d9" (UID: "549f7592-f9b1-45a9-a1af-3bfd3964e3d9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:26:55.053572 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:26:55.053545    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/549f7592-f9b1-45a9-a1af-3bfd3964e3d9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "549f7592-f9b1-45a9-a1af-3bfd3964e3d9" (UID: "549f7592-f9b1-45a9-a1af-3bfd3964e3d9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:26:55.053572 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:26:55.053552    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/549f7592-f9b1-45a9-a1af-3bfd3964e3d9-dshm" (OuterVolumeSpecName: "dshm") pod "549f7592-f9b1-45a9-a1af-3bfd3964e3d9" (UID: "549f7592-f9b1-45a9-a1af-3bfd3964e3d9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:26:55.053701 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:26:55.053596    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/549f7592-f9b1-45a9-a1af-3bfd3964e3d9-kube-api-access-sbg4x" (OuterVolumeSpecName: "kube-api-access-sbg4x") pod "549f7592-f9b1-45a9-a1af-3bfd3964e3d9" (UID: "549f7592-f9b1-45a9-a1af-3bfd3964e3d9"). InnerVolumeSpecName "kube-api-access-sbg4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:26:55.153180 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:26:55.153146    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/549f7592-f9b1-45a9-a1af-3bfd3964e3d9-tls-certs\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:26:55.153180 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:26:55.153175    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/549f7592-f9b1-45a9-a1af-3bfd3964e3d9-model-cache\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:26:55.153180 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:26:55.153183    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/549f7592-f9b1-45a9-a1af-3bfd3964e3d9-home\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:26:55.153408 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:26:55.153192    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbg4x\" (UniqueName: \"kubernetes.io/projected/549f7592-f9b1-45a9-a1af-3bfd3964e3d9-kube-api-access-sbg4x\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:26:55.153408 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:26:55.153202    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/549f7592-f9b1-45a9-a1af-3bfd3964e3d9-dshm\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:26:55.153408 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:26:55.153210    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/549f7592-f9b1-45a9-a1af-3bfd3964e3d9-tmp-dir\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:26:55.493987 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:26:55.493956    2575 generic.go:356] "Generic (PLEG): container finished" podID="549f7592-f9b1-45a9-a1af-3bfd3964e3d9" containerID="07660cf5542f9133c13ad1973fcd45228d6d7ba6f78e0711b9e616f50c516ecf" exitCode=0
Sep 17 21:26:55.494425 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:26:55.494000    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sk228"
Sep 17 21:26:55.494425 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:26:55.494032    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sk228" event={"ID":"549f7592-f9b1-45a9-a1af-3bfd3964e3d9","Type":"ContainerDied","Data":"07660cf5542f9133c13ad1973fcd45228d6d7ba6f78e0711b9e616f50c516ecf"}
Sep 17 21:26:55.494425 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:26:55.494099    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sk228" event={"ID":"549f7592-f9b1-45a9-a1af-3bfd3964e3d9","Type":"ContainerDied","Data":"d7a74d3426498df96768d34c0ad0b370b0efdcbae8b0af1bcc6a1b7a0096d911"}
Sep 17 21:26:55.494425 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:26:55.494136    2575 scope.go:117] "RemoveContainer" containerID="07660cf5542f9133c13ad1973fcd45228d6d7ba6f78e0711b9e616f50c516ecf"
Sep 17 21:26:55.503180 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:26:55.503163    2575 scope.go:117] "RemoveContainer" containerID="07660cf5542f9133c13ad1973fcd45228d6d7ba6f78e0711b9e616f50c516ecf"
Sep 17 21:26:55.503473 ip-10-0-139-36 kubenswrapper[2575]: E0917 21:26:55.503452    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07660cf5542f9133c13ad1973fcd45228d6d7ba6f78e0711b9e616f50c516ecf\": container with ID starting with 07660cf5542f9133c13ad1973fcd45228d6d7ba6f78e0711b9e616f50c516ecf not found: ID does not exist" containerID="07660cf5542f9133c13ad1973fcd45228d6d7ba6f78e0711b9e616f50c516ecf"
Sep 17 21:26:55.503559 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:26:55.503486    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07660cf5542f9133c13ad1973fcd45228d6d7ba6f78e0711b9e616f50c516ecf"} err="failed to get container status \"07660cf5542f9133c13ad1973fcd45228d6d7ba6f78e0711b9e616f50c516ecf\": rpc error: code = NotFound desc = could not find container \"07660cf5542f9133c13ad1973fcd45228d6d7ba6f78e0711b9e616f50c516ecf\": container with ID starting with 07660cf5542f9133c13ad1973fcd45228d6d7ba6f78e0711b9e616f50c516ecf not found: ID does not exist"
Sep 17 21:26:55.513287 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:26:55.513266    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sk228"]
Sep 17 21:26:55.516295 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:26:55.516276    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sk228"]
Sep 17 21:26:55.694230 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:26:55.694153    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:26:57.356899 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:26:57.356867    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="549f7592-f9b1-45a9-a1af-3bfd3964e3d9" path="/var/lib/kubelet/pods/549f7592-f9b1-45a9-a1af-3bfd3964e3d9/volumes"
Sep 17 21:27:43.709407 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:27:43.709348    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:27:43.808500 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:27:43.808466    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:27:44.445786 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:27:44.445745    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:27:44.486563 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:27:44.486531    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796967nn2"]
Sep 17 21:27:44.497620 ip-10-0-139-36 kubenswrapper[2575]: E0917 21:27:44.497580    2575 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 21:27:44.497620 ip-10-0-139-36 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:27:44.497620 ip-10-0-139-36 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 17 21:27:44.497620 ip-10-0-139-36 kubenswrapper[2575]:  > containerID="0004657109a1b0e9d9b58f0e5e6eb437723820348741d3165710c82c8d5f3077" cmd=["/bin/sleep","15"]
Sep 17 21:27:44.497961 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:27:44.497636    2575 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796967nn2" message="" err=<
Sep 17 21:27:44.497961 ip-10-0-139-36 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:27:44.497961 ip-10-0-139-36 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 17 21:27:44.497961 ip-10-0-139-36 kubenswrapper[2575]:  >
Sep 17 21:27:44.497961 ip-10-0-139-36 kubenswrapper[2575]: E0917 21:27:44.497648    2575 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 21:27:44.497961 ip-10-0-139-36 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:27:44.497961 ip-10-0-139-36 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 17 21:27:44.497961 ip-10-0-139-36 kubenswrapper[2575]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796967nn2" podUID="69e49267-cd70-4617-bfa2-f4bc95ed2598" containerName="main" containerID="cri-o://0004657109a1b0e9d9b58f0e5e6eb437723820348741d3165710c82c8d5f3077"
Sep 17 21:27:44.497961 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:27:44.497719    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796967nn2" podUID="69e49267-cd70-4617-bfa2-f4bc95ed2598" containerName="main" containerID="cri-o://0004657109a1b0e9d9b58f0e5e6eb437723820348741d3165710c82c8d5f3077" gracePeriod=60
Sep 17 21:27:44.564056 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:27:44.564018    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:27:44.738475 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:27:44.738448    2575 generic.go:356] "Generic (PLEG): container finished" podID="69e49267-cd70-4617-bfa2-f4bc95ed2598" containerID="0004657109a1b0e9d9b58f0e5e6eb437723820348741d3165710c82c8d5f3077" exitCode=0
Sep 17 21:27:44.738810 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:27:44.738528    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796967nn2" event={"ID":"69e49267-cd70-4617-bfa2-f4bc95ed2598","Type":"ContainerDied","Data":"0004657109a1b0e9d9b58f0e5e6eb437723820348741d3165710c82c8d5f3077"}
Sep 17 21:27:44.838063 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:27:44.838040    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796967nn2"
Sep 17 21:27:44.885191 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:27:44.885166    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69e49267-cd70-4617-bfa2-f4bc95ed2598-model-cache\") pod \"69e49267-cd70-4617-bfa2-f4bc95ed2598\" (UID: \"69e49267-cd70-4617-bfa2-f4bc95ed2598\") "
Sep 17 21:27:44.885317 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:27:44.885202    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69e49267-cd70-4617-bfa2-f4bc95ed2598-tmp-dir\") pod \"69e49267-cd70-4617-bfa2-f4bc95ed2598\" (UID: \"69e49267-cd70-4617-bfa2-f4bc95ed2598\") "
Sep 17 21:27:44.885317 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:27:44.885223    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69e49267-cd70-4617-bfa2-f4bc95ed2598-home\") pod \"69e49267-cd70-4617-bfa2-f4bc95ed2598\" (UID: \"69e49267-cd70-4617-bfa2-f4bc95ed2598\") "
Sep 17 21:27:44.885317 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:27:44.885283    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-njbnn\" (UniqueName: \"kubernetes.io/projected/69e49267-cd70-4617-bfa2-f4bc95ed2598-kube-api-access-njbnn\") pod \"69e49267-cd70-4617-bfa2-f4bc95ed2598\" (UID: \"69e49267-cd70-4617-bfa2-f4bc95ed2598\") "
Sep 17 21:27:44.885317 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:27:44.885313    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69e49267-cd70-4617-bfa2-f4bc95ed2598-dshm\") pod \"69e49267-cd70-4617-bfa2-f4bc95ed2598\" (UID: \"69e49267-cd70-4617-bfa2-f4bc95ed2598\") "
Sep 17 21:27:44.885543 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:27:44.885416    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69e49267-cd70-4617-bfa2-f4bc95ed2598-tls-certs\") pod \"69e49267-cd70-4617-bfa2-f4bc95ed2598\" (UID: \"69e49267-cd70-4617-bfa2-f4bc95ed2598\") "
Sep 17 21:27:44.885543 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:27:44.885475    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69e49267-cd70-4617-bfa2-f4bc95ed2598-model-cache" (OuterVolumeSpecName: "model-cache") pod "69e49267-cd70-4617-bfa2-f4bc95ed2598" (UID: "69e49267-cd70-4617-bfa2-f4bc95ed2598"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:27:44.885543 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:27:44.885491    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69e49267-cd70-4617-bfa2-f4bc95ed2598-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "69e49267-cd70-4617-bfa2-f4bc95ed2598" (UID: "69e49267-cd70-4617-bfa2-f4bc95ed2598"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:27:44.885700 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:27:44.885596    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69e49267-cd70-4617-bfa2-f4bc95ed2598-home" (OuterVolumeSpecName: "home") pod "69e49267-cd70-4617-bfa2-f4bc95ed2598" (UID: "69e49267-cd70-4617-bfa2-f4bc95ed2598"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:27:44.885770 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:27:44.885752    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69e49267-cd70-4617-bfa2-f4bc95ed2598-model-cache\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:27:44.885820 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:27:44.885776    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69e49267-cd70-4617-bfa2-f4bc95ed2598-tmp-dir\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:27:44.885820 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:27:44.885790    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69e49267-cd70-4617-bfa2-f4bc95ed2598-home\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:27:44.887420 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:27:44.887398    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69e49267-cd70-4617-bfa2-f4bc95ed2598-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "69e49267-cd70-4617-bfa2-f4bc95ed2598" (UID: "69e49267-cd70-4617-bfa2-f4bc95ed2598"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:27:44.887557 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:27:44.887471    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69e49267-cd70-4617-bfa2-f4bc95ed2598-kube-api-access-njbnn" (OuterVolumeSpecName: "kube-api-access-njbnn") pod "69e49267-cd70-4617-bfa2-f4bc95ed2598" (UID: "69e49267-cd70-4617-bfa2-f4bc95ed2598"). InnerVolumeSpecName "kube-api-access-njbnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:27:44.887791 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:27:44.887774    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69e49267-cd70-4617-bfa2-f4bc95ed2598-dshm" (OuterVolumeSpecName: "dshm") pod "69e49267-cd70-4617-bfa2-f4bc95ed2598" (UID: "69e49267-cd70-4617-bfa2-f4bc95ed2598"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:27:44.986217 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:27:44.986142    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njbnn\" (UniqueName: \"kubernetes.io/projected/69e49267-cd70-4617-bfa2-f4bc95ed2598-kube-api-access-njbnn\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:27:44.986217 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:27:44.986168    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69e49267-cd70-4617-bfa2-f4bc95ed2598-dshm\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:27:44.986217 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:27:44.986181    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69e49267-cd70-4617-bfa2-f4bc95ed2598-tls-certs\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:27:45.744830 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:27:45.744797    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796967nn2" event={"ID":"69e49267-cd70-4617-bfa2-f4bc95ed2598","Type":"ContainerDied","Data":"363492a8e043f0c629ad755702aa4c36aa20a0c6750752d7e4f2d3dd87259fa4"}
Sep 17 21:27:45.744830 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:27:45.744817    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796967nn2"
Sep 17 21:27:45.745366 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:27:45.744892    2575 scope.go:117] "RemoveContainer" containerID="0004657109a1b0e9d9b58f0e5e6eb437723820348741d3165710c82c8d5f3077"
Sep 17 21:27:45.763888 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:27:45.763836    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796967nn2"]
Sep 17 21:27:45.770491 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:27:45.770470    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796967nn2"]
Sep 17 21:27:47.357005 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:27:47.356971    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69e49267-cd70-4617-bfa2-f4bc95ed2598" path="/var/lib/kubelet/pods/69e49267-cd70-4617-bfa2-f4bc95ed2598/volumes"
Sep 17 21:27:50.389361 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:27:50.389283    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:27:52.293028 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:27:52.292995    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:27:52.538882 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:27:52.538831    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:27:56.541924 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:27:56.541886    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:27:56.646290 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:27:56.646261    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:27:59.149692 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:27:59.149645    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:28:34.253439 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:28:34.253405    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:28:34.351891 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:28:34.351837    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:28:40.875678 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:28:40.875641    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:28:44.331392 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:28:44.331354    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx"]
Sep 17 21:28:44.331919 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:28:44.331719    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69e49267-cd70-4617-bfa2-f4bc95ed2598" containerName="main"
Sep 17 21:28:44.331919 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:28:44.331730    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="69e49267-cd70-4617-bfa2-f4bc95ed2598" containerName="main"
Sep 17 21:28:44.331919 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:28:44.331739    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="549f7592-f9b1-45a9-a1af-3bfd3964e3d9" containerName="main"
Sep 17 21:28:44.331919 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:28:44.331745    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="549f7592-f9b1-45a9-a1af-3bfd3964e3d9" containerName="main"
Sep 17 21:28:44.331919 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:28:44.331805    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="69e49267-cd70-4617-bfa2-f4bc95ed2598" containerName="main"
Sep 17 21:28:44.331919 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:28:44.331816    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="549f7592-f9b1-45a9-a1af-3bfd3964e3d9" containerName="main"
Sep 17 21:28:44.334964 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:28:44.334943    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx"
Sep 17 21:28:44.338013 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:28:44.337992    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"seaweedfs-s3-creds\""
Sep 17 21:28:44.338408 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:28:44.338375    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Sep 17 21:28:44.338615 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:28:44.338577    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 21:28:44.338615 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:28:44.338593    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Sep 17 21:28:44.338797 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:28:44.338632    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Sep 17 21:28:44.339015 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:28:44.338996    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-b25p2\""
Sep 17 21:28:44.351618 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:28:44.351597    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx"]
Sep 17 21:28:44.368944 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:28:44.368918    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:28:44.400699 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:28:44.400674    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl7pv\" (UniqueName: \"kubernetes.io/projected/e42f9299-ed56-4bb9-9cac-3675073e58b7-kube-api-access-jl7pv\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx\" (UID: \"e42f9299-ed56-4bb9-9cac-3675073e58b7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx"
Sep 17 21:28:44.400818 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:28:44.400710    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e42f9299-ed56-4bb9-9cac-3675073e58b7-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx\" (UID: \"e42f9299-ed56-4bb9-9cac-3675073e58b7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx"
Sep 17 21:28:44.400818 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:28:44.400735    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e42f9299-ed56-4bb9-9cac-3675073e58b7-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx\" (UID: \"e42f9299-ed56-4bb9-9cac-3675073e58b7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx"
Sep 17 21:28:44.400818 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:28:44.400777    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e42f9299-ed56-4bb9-9cac-3675073e58b7-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx\" (UID: \"e42f9299-ed56-4bb9-9cac-3675073e58b7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx"
Sep 17 21:28:44.400818 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:28:44.400800    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e42f9299-ed56-4bb9-9cac-3675073e58b7-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx\" (UID: \"e42f9299-ed56-4bb9-9cac-3675073e58b7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx"
Sep 17 21:28:44.401022 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:28:44.400921    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e42f9299-ed56-4bb9-9cac-3675073e58b7-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx\" (UID: \"e42f9299-ed56-4bb9-9cac-3675073e58b7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx"
Sep 17 21:28:44.401022 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:28:44.400958    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e42f9299-ed56-4bb9-9cac-3675073e58b7-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx\" (UID: \"e42f9299-ed56-4bb9-9cac-3675073e58b7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx"
Sep 17 21:28:44.401104 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:28:44.401065    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e42f9299-ed56-4bb9-9cac-3675073e58b7-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx\" (UID: \"e42f9299-ed56-4bb9-9cac-3675073e58b7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx"
Sep 17 21:28:44.502051 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:28:44.502021    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e42f9299-ed56-4bb9-9cac-3675073e58b7-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx\" (UID: \"e42f9299-ed56-4bb9-9cac-3675073e58b7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx"
Sep 17 21:28:44.502217 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:28:44.502064    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl7pv\" (UniqueName: \"kubernetes.io/projected/e42f9299-ed56-4bb9-9cac-3675073e58b7-kube-api-access-jl7pv\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx\" (UID: \"e42f9299-ed56-4bb9-9cac-3675073e58b7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx"
Sep 17 21:28:44.502217 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:28:44.502100    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e42f9299-ed56-4bb9-9cac-3675073e58b7-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx\" (UID: \"e42f9299-ed56-4bb9-9cac-3675073e58b7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx"
Sep 17 21:28:44.502217 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:28:44.502131    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e42f9299-ed56-4bb9-9cac-3675073e58b7-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx\" (UID: \"e42f9299-ed56-4bb9-9cac-3675073e58b7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx"
Sep 17 21:28:44.502217 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:28:44.502163    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e42f9299-ed56-4bb9-9cac-3675073e58b7-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx\" (UID: \"e42f9299-ed56-4bb9-9cac-3675073e58b7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx"
Sep 17 21:28:44.502217 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:28:44.502193    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e42f9299-ed56-4bb9-9cac-3675073e58b7-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx\" (UID: \"e42f9299-ed56-4bb9-9cac-3675073e58b7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx"
Sep 17 21:28:44.502490 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:28:44.502236    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e42f9299-ed56-4bb9-9cac-3675073e58b7-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx\" (UID: \"e42f9299-ed56-4bb9-9cac-3675073e58b7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx"
Sep 17 21:28:44.502490 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:28:44.502264    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e42f9299-ed56-4bb9-9cac-3675073e58b7-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx\" (UID: \"e42f9299-ed56-4bb9-9cac-3675073e58b7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx"
Sep 17 21:28:44.502608 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:28:44.502576    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e42f9299-ed56-4bb9-9cac-3675073e58b7-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx\" (UID: \"e42f9299-ed56-4bb9-9cac-3675073e58b7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx"
Sep 17 21:28:44.502754 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:28:44.502733    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e42f9299-ed56-4bb9-9cac-3675073e58b7-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx\" (UID: \"e42f9299-ed56-4bb9-9cac-3675073e58b7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx"
Sep 17 21:28:44.502839 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:28:44.502769    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e42f9299-ed56-4bb9-9cac-3675073e58b7-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx\" (UID: \"e42f9299-ed56-4bb9-9cac-3675073e58b7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx"
Sep 17 21:28:44.502839 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:28:44.502806    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e42f9299-ed56-4bb9-9cac-3675073e58b7-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx\" (UID: \"e42f9299-ed56-4bb9-9cac-3675073e58b7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx"
Sep 17 21:28:44.503041 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:28:44.503016    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e42f9299-ed56-4bb9-9cac-3675073e58b7-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx\" (UID: \"e42f9299-ed56-4bb9-9cac-3675073e58b7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx"
Sep 17 21:28:44.504449 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:28:44.504421    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e42f9299-ed56-4bb9-9cac-3675073e58b7-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx\" (UID: \"e42f9299-ed56-4bb9-9cac-3675073e58b7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx"
Sep 17 21:28:44.504726 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:28:44.504708    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e42f9299-ed56-4bb9-9cac-3675073e58b7-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx\" (UID: \"e42f9299-ed56-4bb9-9cac-3675073e58b7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx"
Sep 17 21:28:44.511004 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:28:44.510983    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl7pv\" (UniqueName: \"kubernetes.io/projected/e42f9299-ed56-4bb9-9cac-3675073e58b7-kube-api-access-jl7pv\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx\" (UID: \"e42f9299-ed56-4bb9-9cac-3675073e58b7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx"
Sep 17 21:28:44.608598 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:28:44.608517    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:28:44.645243 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:28:44.645215    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx"
Sep 17 21:28:44.925381 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:28:44.925300    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx"]
Sep 17 21:28:44.929131 ip-10-0-139-36 kubenswrapper[2575]: W0917 21:28:44.929101    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode42f9299_ed56_4bb9_9cac_3675073e58b7.slice/crio-3219a8b766112719b842295799d64d1d412b6698c3ef83b2f5c7442caa33bfa7 WatchSource:0}: Error finding container 3219a8b766112719b842295799d64d1d412b6698c3ef83b2f5c7442caa33bfa7: Status 404 returned error can't find the container with id 3219a8b766112719b842295799d64d1d412b6698c3ef83b2f5c7442caa33bfa7
Sep 17 21:28:45.039349 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:28:45.039305    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx" event={"ID":"e42f9299-ed56-4bb9-9cac-3675073e58b7","Type":"ContainerStarted","Data":"96ff0ba7351ee92581af5f2c806ac82d2fce891455b650e19c3d372a284a6e66"}
Sep 17 21:28:45.039545 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:28:45.039379    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx" event={"ID":"e42f9299-ed56-4bb9-9cac-3675073e58b7","Type":"ContainerStarted","Data":"3219a8b766112719b842295799d64d1d412b6698c3ef83b2f5c7442caa33bfa7"}
Sep 17 21:28:47.427713 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:28:47.427676    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:28:47.502862 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:28:47.502805    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:28:48.055681 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:28:48.055645    2575 generic.go:356] "Generic (PLEG): container finished" podID="e42f9299-ed56-4bb9-9cac-3675073e58b7" containerID="96ff0ba7351ee92581af5f2c806ac82d2fce891455b650e19c3d372a284a6e66" exitCode=0
Sep 17 21:28:48.055844 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:28:48.055718    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx" event={"ID":"e42f9299-ed56-4bb9-9cac-3675073e58b7","Type":"ContainerDied","Data":"96ff0ba7351ee92581af5f2c806ac82d2fce891455b650e19c3d372a284a6e66"}
Sep 17 21:28:49.062117 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:28:49.062084    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx" event={"ID":"e42f9299-ed56-4bb9-9cac-3675073e58b7","Type":"ContainerStarted","Data":"ef93eb4aa2b4ba42547bc504a5cb910fe74b6929efbb7d22ad3b0da92d19390c"}
Sep 17 21:28:49.086743 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:28:49.086690    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx" podStartSLOduration=2.434119141 podStartE2EDuration="5.086674256s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.652555115s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:28:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:28:49.083718522 +0000 UTC m=+2394.236162898" watchObservedRunningTime="2026-09-17 21:28:49.086674256 +0000 UTC m=+2394.239118683"
Sep 17 21:28:49.645589 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:28:49.645554    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx"
Sep 17 21:28:54.646284 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:28:54.646248    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx"
Sep 17 21:28:54.648092 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:28:54.648061    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx" podUID="e42f9299-ed56-4bb9-9cac-3675073e58b7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 17 21:29:04.646178 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:29:04.646126    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx" podUID="e42f9299-ed56-4bb9-9cac-3675073e58b7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 17 21:29:12.257199 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:29:12.257157    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:29:14.645739 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:29:14.645697    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx" podUID="e42f9299-ed56-4bb9-9cac-3675073e58b7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 17 21:29:24.646635 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:29:24.646540    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx" podUID="e42f9299-ed56-4bb9-9cac-3675073e58b7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 17 21:29:34.646392 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:29:34.646345    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx" podUID="e42f9299-ed56-4bb9-9cac-3675073e58b7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 17 21:29:44.646637 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:29:44.646592    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx" podUID="e42f9299-ed56-4bb9-9cac-3675073e58b7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 17 21:29:54.646643 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:29:54.646587    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx" podUID="e42f9299-ed56-4bb9-9cac-3675073e58b7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 17 21:29:56.980443 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:29:56.980394    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:29:57.047622 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:29:57.047586    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:30:04.656238 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:30:04.656206    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx"
Sep 17 21:30:04.664188 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:30:04.664162    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx"
Sep 17 21:30:31.773237 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:30:31.773205    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:30:31.799920 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:30:31.799884    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx"]
Sep 17 21:30:31.898765 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:30:31.898714    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:30:46.824761 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:30:46.824718    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx" podUID="e42f9299-ed56-4bb9-9cac-3675073e58b7" containerName="main" containerID="cri-o://ef93eb4aa2b4ba42547bc504a5cb910fe74b6929efbb7d22ad3b0da92d19390c" gracePeriod=45
Sep 17 21:30:47.249048 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:30:47.249001    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:30:47.481563 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:30:47.481534    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx"
Sep 17 21:30:47.564393 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:30:47.564314    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:30:47.640801 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:30:47.640768    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e42f9299-ed56-4bb9-9cac-3675073e58b7-model-cache\") pod \"e42f9299-ed56-4bb9-9cac-3675073e58b7\" (UID: \"e42f9299-ed56-4bb9-9cac-3675073e58b7\") "
Sep 17 21:30:47.641008 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:30:47.640809    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e42f9299-ed56-4bb9-9cac-3675073e58b7-home\") pod \"e42f9299-ed56-4bb9-9cac-3675073e58b7\" (UID: \"e42f9299-ed56-4bb9-9cac-3675073e58b7\") "
Sep 17 21:30:47.641008 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:30:47.640832    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jl7pv\" (UniqueName: \"kubernetes.io/projected/e42f9299-ed56-4bb9-9cac-3675073e58b7-kube-api-access-jl7pv\") pod \"e42f9299-ed56-4bb9-9cac-3675073e58b7\" (UID: \"e42f9299-ed56-4bb9-9cac-3675073e58b7\") "
Sep 17 21:30:47.641008 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:30:47.640864    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e42f9299-ed56-4bb9-9cac-3675073e58b7-tls-certs\") pod \"e42f9299-ed56-4bb9-9cac-3675073e58b7\" (UID: \"e42f9299-ed56-4bb9-9cac-3675073e58b7\") "
Sep 17 21:30:47.641182 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:30:47.641128    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e42f9299-ed56-4bb9-9cac-3675073e58b7-cabundle-cert\") pod \"e42f9299-ed56-4bb9-9cac-3675073e58b7\" (UID: \"e42f9299-ed56-4bb9-9cac-3675073e58b7\") "
Sep 17 21:30:47.641182 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:30:47.641130    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e42f9299-ed56-4bb9-9cac-3675073e58b7-model-cache" (OuterVolumeSpecName: "model-cache") pod "e42f9299-ed56-4bb9-9cac-3675073e58b7" (UID: "e42f9299-ed56-4bb9-9cac-3675073e58b7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:30:47.641292 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:30:47.641219    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e42f9299-ed56-4bb9-9cac-3675073e58b7-dshm\") pod \"e42f9299-ed56-4bb9-9cac-3675073e58b7\" (UID: \"e42f9299-ed56-4bb9-9cac-3675073e58b7\") "
Sep 17 21:30:47.641292 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:30:47.641241    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e42f9299-ed56-4bb9-9cac-3675073e58b7-home" (OuterVolumeSpecName: "home") pod "e42f9299-ed56-4bb9-9cac-3675073e58b7" (UID: "e42f9299-ed56-4bb9-9cac-3675073e58b7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:30:47.641404 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:30:47.641288    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e42f9299-ed56-4bb9-9cac-3675073e58b7-tmp-dir\") pod \"e42f9299-ed56-4bb9-9cac-3675073e58b7\" (UID: \"e42f9299-ed56-4bb9-9cac-3675073e58b7\") "
Sep 17 21:30:47.641404 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:30:47.641365    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e42f9299-ed56-4bb9-9cac-3675073e58b7-kserve-provision-location\") pod \"e42f9299-ed56-4bb9-9cac-3675073e58b7\" (UID: \"e42f9299-ed56-4bb9-9cac-3675073e58b7\") "
Sep 17 21:30:47.641507 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:30:47.641452    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e42f9299-ed56-4bb9-9cac-3675073e58b7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e42f9299-ed56-4bb9-9cac-3675073e58b7" (UID: "e42f9299-ed56-4bb9-9cac-3675073e58b7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:30:47.641711 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:30:47.641690    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e42f9299-ed56-4bb9-9cac-3675073e58b7-model-cache\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:30:47.641890 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:30:47.641869    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e42f9299-ed56-4bb9-9cac-3675073e58b7-home\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:30:47.641995 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:30:47.641892    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e42f9299-ed56-4bb9-9cac-3675073e58b7-cabundle-cert\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:30:47.643796 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:30:47.643752    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e42f9299-ed56-4bb9-9cac-3675073e58b7-dshm" (OuterVolumeSpecName: "dshm") pod "e42f9299-ed56-4bb9-9cac-3675073e58b7" (UID: "e42f9299-ed56-4bb9-9cac-3675073e58b7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:30:47.643796 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:30:47.643757    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e42f9299-ed56-4bb9-9cac-3675073e58b7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e42f9299-ed56-4bb9-9cac-3675073e58b7" (UID: "e42f9299-ed56-4bb9-9cac-3675073e58b7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:30:47.644073 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:30:47.643934    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e42f9299-ed56-4bb9-9cac-3675073e58b7-kube-api-access-jl7pv" (OuterVolumeSpecName: "kube-api-access-jl7pv") pod "e42f9299-ed56-4bb9-9cac-3675073e58b7" (UID: "e42f9299-ed56-4bb9-9cac-3675073e58b7"). InnerVolumeSpecName "kube-api-access-jl7pv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:30:47.655488 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:30:47.655457    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e42f9299-ed56-4bb9-9cac-3675073e58b7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e42f9299-ed56-4bb9-9cac-3675073e58b7" (UID: "e42f9299-ed56-4bb9-9cac-3675073e58b7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:30:47.669054 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:30:47.669023    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e42f9299-ed56-4bb9-9cac-3675073e58b7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e42f9299-ed56-4bb9-9cac-3675073e58b7" (UID: "e42f9299-ed56-4bb9-9cac-3675073e58b7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:30:47.688905 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:30:47.688875    2575 generic.go:356] "Generic (PLEG): container finished" podID="e42f9299-ed56-4bb9-9cac-3675073e58b7" containerID="ef93eb4aa2b4ba42547bc504a5cb910fe74b6929efbb7d22ad3b0da92d19390c" exitCode=0
Sep 17 21:30:47.689055 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:30:47.688935    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx"
Sep 17 21:30:47.689055 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:30:47.688969    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx" event={"ID":"e42f9299-ed56-4bb9-9cac-3675073e58b7","Type":"ContainerDied","Data":"ef93eb4aa2b4ba42547bc504a5cb910fe74b6929efbb7d22ad3b0da92d19390c"}
Sep 17 21:30:47.689055 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:30:47.689030    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx" event={"ID":"e42f9299-ed56-4bb9-9cac-3675073e58b7","Type":"ContainerDied","Data":"3219a8b766112719b842295799d64d1d412b6698c3ef83b2f5c7442caa33bfa7"}
Sep 17 21:30:47.689055 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:30:47.689055    2575 scope.go:117] "RemoveContainer" containerID="ef93eb4aa2b4ba42547bc504a5cb910fe74b6929efbb7d22ad3b0da92d19390c"
Sep 17 21:30:47.698763 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:30:47.698744    2575 scope.go:117] "RemoveContainer" containerID="96ff0ba7351ee92581af5f2c806ac82d2fce891455b650e19c3d372a284a6e66"
Sep 17 21:30:47.707389 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:30:47.707373    2575 scope.go:117] "RemoveContainer" containerID="ef93eb4aa2b4ba42547bc504a5cb910fe74b6929efbb7d22ad3b0da92d19390c"
Sep 17 21:30:47.707671 ip-10-0-139-36 kubenswrapper[2575]: E0917 21:30:47.707648    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef93eb4aa2b4ba42547bc504a5cb910fe74b6929efbb7d22ad3b0da92d19390c\": container with ID starting with ef93eb4aa2b4ba42547bc504a5cb910fe74b6929efbb7d22ad3b0da92d19390c not found: ID does not exist" containerID="ef93eb4aa2b4ba42547bc504a5cb910fe74b6929efbb7d22ad3b0da92d19390c"
Sep 17 21:30:47.707755 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:30:47.707677    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef93eb4aa2b4ba42547bc504a5cb910fe74b6929efbb7d22ad3b0da92d19390c"} err="failed to get container status \"ef93eb4aa2b4ba42547bc504a5cb910fe74b6929efbb7d22ad3b0da92d19390c\": rpc error: code = NotFound desc = could not find container \"ef93eb4aa2b4ba42547bc504a5cb910fe74b6929efbb7d22ad3b0da92d19390c\": container with ID starting with ef93eb4aa2b4ba42547bc504a5cb910fe74b6929efbb7d22ad3b0da92d19390c not found: ID does not exist"
Sep 17 21:30:47.707755 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:30:47.707694    2575 scope.go:117] "RemoveContainer" containerID="96ff0ba7351ee92581af5f2c806ac82d2fce891455b650e19c3d372a284a6e66"
Sep 17 21:30:47.707993 ip-10-0-139-36 kubenswrapper[2575]: E0917 21:30:47.707971    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96ff0ba7351ee92581af5f2c806ac82d2fce891455b650e19c3d372a284a6e66\": container with ID starting with 96ff0ba7351ee92581af5f2c806ac82d2fce891455b650e19c3d372a284a6e66 not found: ID does not exist" containerID="96ff0ba7351ee92581af5f2c806ac82d2fce891455b650e19c3d372a284a6e66"
Sep 17 21:30:47.708066 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:30:47.708003    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96ff0ba7351ee92581af5f2c806ac82d2fce891455b650e19c3d372a284a6e66"} err="failed to get container status \"96ff0ba7351ee92581af5f2c806ac82d2fce891455b650e19c3d372a284a6e66\": rpc error: code = NotFound desc = could not find container \"96ff0ba7351ee92581af5f2c806ac82d2fce891455b650e19c3d372a284a6e66\": container with ID starting with 96ff0ba7351ee92581af5f2c806ac82d2fce891455b650e19c3d372a284a6e66 not found: ID does not exist"
Sep 17 21:30:47.736284 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:30:47.736230    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx"]
Sep 17 21:30:47.743256 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:30:47.743231    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jl7pv\" (UniqueName: \"kubernetes.io/projected/e42f9299-ed56-4bb9-9cac-3675073e58b7-kube-api-access-jl7pv\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:30:47.743373 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:30:47.743259    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e42f9299-ed56-4bb9-9cac-3675073e58b7-tls-certs\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:30:47.743373 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:30:47.743276    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e42f9299-ed56-4bb9-9cac-3675073e58b7-dshm\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:30:47.743373 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:30:47.743288    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e42f9299-ed56-4bb9-9cac-3675073e58b7-tmp-dir\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:30:47.743373 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:30:47.743299    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e42f9299-ed56-4bb9-9cac-3675073e58b7-kserve-provision-location\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:30:47.747020 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:30:47.746994    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5bb765687c-c7rsx"]
Sep 17 21:30:47.771809 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:30:47.771780    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:30:47.953103 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:30:47.953075    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:30:49.356865 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:30:49.356819    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e42f9299-ed56-4bb9-9cac-3675073e58b7" path="/var/lib/kubelet/pods/e42f9299-ed56-4bb9-9cac-3675073e58b7/volumes"
Sep 17 21:30:51.189252 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:30:51.189215    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:30:53.892352 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:30:53.892320    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:31:00.463715 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:31:00.463671    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:31:00.559793 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:31:00.559752    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:31:09.313010 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:31:09.312974    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c87b9cbcd-fl5nx"]
Sep 17 21:31:09.313499 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:31:09.313445    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e42f9299-ed56-4bb9-9cac-3675073e58b7" containerName="storage-initializer"
Sep 17 21:31:09.313499 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:31:09.313461    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e42f9299-ed56-4bb9-9cac-3675073e58b7" containerName="storage-initializer"
Sep 17 21:31:09.313499 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:31:09.313485    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e42f9299-ed56-4bb9-9cac-3675073e58b7" containerName="main"
Sep 17 21:31:09.313499 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:31:09.313494    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e42f9299-ed56-4bb9-9cac-3675073e58b7" containerName="main"
Sep 17 21:31:09.313661 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:31:09.313559    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e42f9299-ed56-4bb9-9cac-3675073e58b7" containerName="main"
Sep 17 21:31:09.316732 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:31:09.316716    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c87b9cbcd-fl5nx"
Sep 17 21:31:09.319502 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:31:09.319480    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"seaweedfs-s3-creds\""
Sep 17 21:31:09.319640 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:31:09.319539    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-tq67m\""
Sep 17 21:31:09.319640 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:31:09.319583    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Sep 17 21:31:09.319753 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:31:09.319732    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Sep 17 21:31:09.319807 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:31:09.319751    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-kserve-self-signed-certs\""
Sep 17 21:31:09.321181 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:31:09.321164    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 21:31:09.334923 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:31:09.334899    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c87b9cbcd-fl5nx"]
Sep 17 21:31:09.344647 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:31:09.344616    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:31:09.419749 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:31:09.419710    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d17f830-972b-405b-9a8b-9d824322299a-kserve-provision-location\") pod \"router-with-refs-test-kserve-7c87b9cbcd-fl5nx\" (UID: \"2d17f830-972b-405b-9a8b-9d824322299a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c87b9cbcd-fl5nx"
Sep 17 21:31:09.419749 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:31:09.419756    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d17f830-972b-405b-9a8b-9d824322299a-cabundle-cert\") pod \"router-with-refs-test-kserve-7c87b9cbcd-fl5nx\" (UID: \"2d17f830-972b-405b-9a8b-9d824322299a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c87b9cbcd-fl5nx"
Sep 17 21:31:09.419998 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:31:09.419886    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d17f830-972b-405b-9a8b-9d824322299a-dshm\") pod \"router-with-refs-test-kserve-7c87b9cbcd-fl5nx\" (UID: \"2d17f830-972b-405b-9a8b-9d824322299a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c87b9cbcd-fl5nx"
Sep 17 21:31:09.419998 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:31:09.419918    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d17f830-972b-405b-9a8b-9d824322299a-home\") pod \"router-with-refs-test-kserve-7c87b9cbcd-fl5nx\" (UID: \"2d17f830-972b-405b-9a8b-9d824322299a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c87b9cbcd-fl5nx"
Sep 17 21:31:09.419998 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:31:09.419941    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d17f830-972b-405b-9a8b-9d824322299a-tmp-dir\") pod \"router-with-refs-test-kserve-7c87b9cbcd-fl5nx\" (UID: \"2d17f830-972b-405b-9a8b-9d824322299a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c87b9cbcd-fl5nx"
Sep 17 21:31:09.419998 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:31:09.419969    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs45j\" (UniqueName: \"kubernetes.io/projected/2d17f830-972b-405b-9a8b-9d824322299a-kube-api-access-rs45j\") pod \"router-with-refs-test-kserve-7c87b9cbcd-fl5nx\" (UID: \"2d17f830-972b-405b-9a8b-9d824322299a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c87b9cbcd-fl5nx"
Sep 17 21:31:09.419998 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:31:09.419991    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d17f830-972b-405b-9a8b-9d824322299a-tls-certs\") pod \"router-with-refs-test-kserve-7c87b9cbcd-fl5nx\" (UID: \"2d17f830-972b-405b-9a8b-9d824322299a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c87b9cbcd-fl5nx"
Sep 17 21:31:09.420234 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:31:09.420105    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d17f830-972b-405b-9a8b-9d824322299a-model-cache\") pod \"router-with-refs-test-kserve-7c87b9cbcd-fl5nx\" (UID: \"2d17f830-972b-405b-9a8b-9d824322299a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c87b9cbcd-fl5nx"
Sep 17 21:31:09.520949 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:31:09.520905    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d17f830-972b-405b-9a8b-9d824322299a-dshm\") pod \"router-with-refs-test-kserve-7c87b9cbcd-fl5nx\" (UID: \"2d17f830-972b-405b-9a8b-9d824322299a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c87b9cbcd-fl5nx"
Sep 17 21:31:09.520949 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:31:09.520951    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d17f830-972b-405b-9a8b-9d824322299a-home\") pod \"router-with-refs-test-kserve-7c87b9cbcd-fl5nx\" (UID: \"2d17f830-972b-405b-9a8b-9d824322299a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c87b9cbcd-fl5nx"
Sep 17 21:31:09.521216 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:31:09.520977    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d17f830-972b-405b-9a8b-9d824322299a-tmp-dir\") pod \"router-with-refs-test-kserve-7c87b9cbcd-fl5nx\" (UID: \"2d17f830-972b-405b-9a8b-9d824322299a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c87b9cbcd-fl5nx"
Sep 17 21:31:09.521216 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:31:09.521017    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs45j\" (UniqueName: \"kubernetes.io/projected/2d17f830-972b-405b-9a8b-9d824322299a-kube-api-access-rs45j\") pod \"router-with-refs-test-kserve-7c87b9cbcd-fl5nx\" (UID: \"2d17f830-972b-405b-9a8b-9d824322299a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c87b9cbcd-fl5nx"
Sep 17 21:31:09.521216 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:31:09.521043    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d17f830-972b-405b-9a8b-9d824322299a-tls-certs\") pod \"router-with-refs-test-kserve-7c87b9cbcd-fl5nx\" (UID: \"2d17f830-972b-405b-9a8b-9d824322299a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c87b9cbcd-fl5nx"
Sep 17 21:31:09.521216 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:31:09.521097    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d17f830-972b-405b-9a8b-9d824322299a-model-cache\") pod \"router-with-refs-test-kserve-7c87b9cbcd-fl5nx\" (UID: \"2d17f830-972b-405b-9a8b-9d824322299a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c87b9cbcd-fl5nx"
Sep 17 21:31:09.521216 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:31:09.521127    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d17f830-972b-405b-9a8b-9d824322299a-kserve-provision-location\") pod \"router-with-refs-test-kserve-7c87b9cbcd-fl5nx\" (UID: \"2d17f830-972b-405b-9a8b-9d824322299a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c87b9cbcd-fl5nx"
Sep 17 21:31:09.521216 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:31:09.521169    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d17f830-972b-405b-9a8b-9d824322299a-cabundle-cert\") pod \"router-with-refs-test-kserve-7c87b9cbcd-fl5nx\" (UID: \"2d17f830-972b-405b-9a8b-9d824322299a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c87b9cbcd-fl5nx"
Sep 17 21:31:09.521533 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:31:09.521370    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d17f830-972b-405b-9a8b-9d824322299a-home\") pod \"router-with-refs-test-kserve-7c87b9cbcd-fl5nx\" (UID: \"2d17f830-972b-405b-9a8b-9d824322299a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c87b9cbcd-fl5nx"
Sep 17 21:31:09.521533 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:31:09.521406    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d17f830-972b-405b-9a8b-9d824322299a-tmp-dir\") pod \"router-with-refs-test-kserve-7c87b9cbcd-fl5nx\" (UID: \"2d17f830-972b-405b-9a8b-9d824322299a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c87b9cbcd-fl5nx"
Sep 17 21:31:09.521668 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:31:09.521647    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d17f830-972b-405b-9a8b-9d824322299a-model-cache\") pod \"router-with-refs-test-kserve-7c87b9cbcd-fl5nx\" (UID: \"2d17f830-972b-405b-9a8b-9d824322299a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c87b9cbcd-fl5nx"
Sep 17 21:31:09.521737 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:31:09.521678    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d17f830-972b-405b-9a8b-9d824322299a-kserve-provision-location\") pod \"router-with-refs-test-kserve-7c87b9cbcd-fl5nx\" (UID: \"2d17f830-972b-405b-9a8b-9d824322299a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c87b9cbcd-fl5nx"
Sep 17 21:31:09.521985 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:31:09.521964    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d17f830-972b-405b-9a8b-9d824322299a-cabundle-cert\") pod \"router-with-refs-test-kserve-7c87b9cbcd-fl5nx\" (UID: \"2d17f830-972b-405b-9a8b-9d824322299a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c87b9cbcd-fl5nx"
Sep 17 21:31:09.523449 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:31:09.523431    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d17f830-972b-405b-9a8b-9d824322299a-dshm\") pod \"router-with-refs-test-kserve-7c87b9cbcd-fl5nx\" (UID: \"2d17f830-972b-405b-9a8b-9d824322299a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c87b9cbcd-fl5nx"
Sep 17 21:31:09.523628 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:31:09.523613    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d17f830-972b-405b-9a8b-9d824322299a-tls-certs\") pod \"router-with-refs-test-kserve-7c87b9cbcd-fl5nx\" (UID: \"2d17f830-972b-405b-9a8b-9d824322299a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c87b9cbcd-fl5nx"
Sep 17 21:31:09.531470 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:31:09.531446    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs45j\" (UniqueName: \"kubernetes.io/projected/2d17f830-972b-405b-9a8b-9d824322299a-kube-api-access-rs45j\") pod \"router-with-refs-test-kserve-7c87b9cbcd-fl5nx\" (UID: \"2d17f830-972b-405b-9a8b-9d824322299a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c87b9cbcd-fl5nx"
Sep 17 21:31:09.627537 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:31:09.627448    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c87b9cbcd-fl5nx"
Sep 17 21:31:09.753035 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:31:09.752994    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:31:09.856606 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:31:09.856565    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:31:09.914103 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:31:09.914070    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c87b9cbcd-fl5nx"]
Sep 17 21:31:09.915323 ip-10-0-139-36 kubenswrapper[2575]: W0917 21:31:09.915297    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d17f830_972b_405b_9a8b_9d824322299a.slice/crio-c7dcb1d0493b22f3451882131180b92c3356e36931a07284fe80fb25780cc33d WatchSource:0}: Error finding container c7dcb1d0493b22f3451882131180b92c3356e36931a07284fe80fb25780cc33d: Status 404 returned error can't find the container with id c7dcb1d0493b22f3451882131180b92c3356e36931a07284fe80fb25780cc33d
Sep 17 21:31:10.807294 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:31:10.807233    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c87b9cbcd-fl5nx" event={"ID":"2d17f830-972b-405b-9a8b-9d824322299a","Type":"ContainerStarted","Data":"18f07eb53a06ce920c6fb3890cb7df2a828c6e1d78562e400dd88b15add677f0"}
Sep 17 21:31:10.807798 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:31:10.807309    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c87b9cbcd-fl5nx" event={"ID":"2d17f830-972b-405b-9a8b-9d824322299a","Type":"ContainerStarted","Data":"c7dcb1d0493b22f3451882131180b92c3356e36931a07284fe80fb25780cc33d"}
Sep 17 21:31:12.819216 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:31:12.819184    2575 generic.go:356] "Generic (PLEG): container finished" podID="2d17f830-972b-405b-9a8b-9d824322299a" containerID="18f07eb53a06ce920c6fb3890cb7df2a828c6e1d78562e400dd88b15add677f0" exitCode=0
Sep 17 21:31:12.819600 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:31:12.819246    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c87b9cbcd-fl5nx" event={"ID":"2d17f830-972b-405b-9a8b-9d824322299a","Type":"ContainerDied","Data":"18f07eb53a06ce920c6fb3890cb7df2a828c6e1d78562e400dd88b15add677f0"}
Sep 17 21:31:13.833975 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:31:13.833934    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c87b9cbcd-fl5nx" event={"ID":"2d17f830-972b-405b-9a8b-9d824322299a","Type":"ContainerStarted","Data":"a50629a2386e1a5b3532c0b249e84205a057117fb709ccd3983c50992a9e761e"}
Sep 17 21:31:13.861506 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:31:13.861446    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c87b9cbcd-fl5nx" podStartSLOduration=2.739904916 podStartE2EDuration="4.861429846s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.12152493s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:31:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:31:13.858701489 +0000 UTC m=+2539.011145889" watchObservedRunningTime="2026-09-17 21:31:13.861429846 +0000 UTC m=+2539.013874222"
Sep 17 21:31:14.628393 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:31:14.628346    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c87b9cbcd-fl5nx"
Sep 17 21:31:19.628103 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:31:19.628060    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c87b9cbcd-fl5nx"
Sep 17 21:31:19.629606 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:31:19.629575    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c87b9cbcd-fl5nx" podUID="2d17f830-972b-405b-9a8b-9d824322299a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Sep 17 21:31:21.656439 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:31:21.656400    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:31:21.748567 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:31:21.748526    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:31:28.326419 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:31:28.326382    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:31:29.628030 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:31:29.627982    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c87b9cbcd-fl5nx" podUID="2d17f830-972b-405b-9a8b-9d824322299a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Sep 17 21:31:30.298784 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:31:30.298740    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:31:34.667184 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:31:34.667142    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:31:34.749512 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:31:34.749473    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:31:39.628825 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:31:39.628780    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c87b9cbcd-fl5nx" podUID="2d17f830-972b-405b-9a8b-9d824322299a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Sep 17 21:31:49.628703 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:31:49.628658    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c87b9cbcd-fl5nx" podUID="2d17f830-972b-405b-9a8b-9d824322299a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Sep 17 21:31:51.862696 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:31:51.862664    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:31:52.138006 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:31:52.137913    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:31:58.986815 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:31:58.986775    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:31:59.628388 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:31:59.628343    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c87b9cbcd-fl5nx" podUID="2d17f830-972b-405b-9a8b-9d824322299a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Sep 17 21:32:09.628246 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:32:09.628199    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c87b9cbcd-fl5nx" podUID="2d17f830-972b-405b-9a8b-9d824322299a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Sep 17 21:32:11.573830 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:32:11.573790    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:32:19.628007 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:32:19.627916    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c87b9cbcd-fl5nx" podUID="2d17f830-972b-405b-9a8b-9d824322299a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Sep 17 21:32:29.628273 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:32:29.628222    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c87b9cbcd-fl5nx" podUID="2d17f830-972b-405b-9a8b-9d824322299a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Sep 17 21:32:39.628430 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:32:39.628384    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c87b9cbcd-fl5nx" podUID="2d17f830-972b-405b-9a8b-9d824322299a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Sep 17 21:32:49.628339 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:32:49.628298    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c87b9cbcd-fl5nx" podUID="2d17f830-972b-405b-9a8b-9d824322299a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Sep 17 21:32:59.628213 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:32:59.628161    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c87b9cbcd-fl5nx" podUID="2d17f830-972b-405b-9a8b-9d824322299a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Sep 17 21:33:09.638701 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:33:09.638667    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c87b9cbcd-fl5nx"
Sep 17 21:33:09.646632 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:33:09.646604    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c87b9cbcd-fl5nx"
Sep 17 21:33:15.276399 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:33:15.276333    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:33:15.283560 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:33:15.283529    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c87b9cbcd-fl5nx"]
Sep 17 21:33:15.355411 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:33:15.355374    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:33:30.305961 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:33:30.305888    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c87b9cbcd-fl5nx" podUID="2d17f830-972b-405b-9a8b-9d824322299a" containerName="main" containerID="cri-o://a50629a2386e1a5b3532c0b249e84205a057117fb709ccd3983c50992a9e761e" gracePeriod=45
Sep 17 21:33:30.639894 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:33:30.639768    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c87b9cbcd-fl5nx" podUID="2d17f830-972b-405b-9a8b-9d824322299a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Sep 17 21:33:30.967752 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:33:30.967729    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c87b9cbcd-fl5nx"
Sep 17 21:33:31.064878 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:33:31.064842    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d17f830-972b-405b-9a8b-9d824322299a-dshm\") pod \"2d17f830-972b-405b-9a8b-9d824322299a\" (UID: \"2d17f830-972b-405b-9a8b-9d824322299a\") "
Sep 17 21:33:31.065058 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:33:31.064905    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d17f830-972b-405b-9a8b-9d824322299a-tmp-dir\") pod \"2d17f830-972b-405b-9a8b-9d824322299a\" (UID: \"2d17f830-972b-405b-9a8b-9d824322299a\") "
Sep 17 21:33:31.065058 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:33:31.064935    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rs45j\" (UniqueName: \"kubernetes.io/projected/2d17f830-972b-405b-9a8b-9d824322299a-kube-api-access-rs45j\") pod \"2d17f830-972b-405b-9a8b-9d824322299a\" (UID: \"2d17f830-972b-405b-9a8b-9d824322299a\") "
Sep 17 21:33:31.065058 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:33:31.064990    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d17f830-972b-405b-9a8b-9d824322299a-home\") pod \"2d17f830-972b-405b-9a8b-9d824322299a\" (UID: \"2d17f830-972b-405b-9a8b-9d824322299a\") "
Sep 17 21:33:31.065058 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:33:31.065047    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d17f830-972b-405b-9a8b-9d824322299a-tls-certs\") pod \"2d17f830-972b-405b-9a8b-9d824322299a\" (UID: \"2d17f830-972b-405b-9a8b-9d824322299a\") "
Sep 17 21:33:31.065296 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:33:31.065104    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d17f830-972b-405b-9a8b-9d824322299a-cabundle-cert\") pod \"2d17f830-972b-405b-9a8b-9d824322299a\" (UID: \"2d17f830-972b-405b-9a8b-9d824322299a\") "
Sep 17 21:33:31.065296 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:33:31.065152    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d17f830-972b-405b-9a8b-9d824322299a-model-cache\") pod \"2d17f830-972b-405b-9a8b-9d824322299a\" (UID: \"2d17f830-972b-405b-9a8b-9d824322299a\") "
Sep 17 21:33:31.065296 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:33:31.065182    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d17f830-972b-405b-9a8b-9d824322299a-kserve-provision-location\") pod \"2d17f830-972b-405b-9a8b-9d824322299a\" (UID: \"2d17f830-972b-405b-9a8b-9d824322299a\") "
Sep 17 21:33:31.065639 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:33:31.065611    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d17f830-972b-405b-9a8b-9d824322299a-home" (OuterVolumeSpecName: "home") pod "2d17f830-972b-405b-9a8b-9d824322299a" (UID: "2d17f830-972b-405b-9a8b-9d824322299a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:33:31.065738 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:33:31.065590    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d17f830-972b-405b-9a8b-9d824322299a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2d17f830-972b-405b-9a8b-9d824322299a" (UID: "2d17f830-972b-405b-9a8b-9d824322299a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:33:31.065808 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:33:31.065775    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d17f830-972b-405b-9a8b-9d824322299a-model-cache" (OuterVolumeSpecName: "model-cache") pod "2d17f830-972b-405b-9a8b-9d824322299a" (UID: "2d17f830-972b-405b-9a8b-9d824322299a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:33:31.067424 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:33:31.067400    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d17f830-972b-405b-9a8b-9d824322299a-kube-api-access-rs45j" (OuterVolumeSpecName: "kube-api-access-rs45j") pod "2d17f830-972b-405b-9a8b-9d824322299a" (UID: "2d17f830-972b-405b-9a8b-9d824322299a"). InnerVolumeSpecName "kube-api-access-rs45j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:33:31.067552 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:33:31.067516    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d17f830-972b-405b-9a8b-9d824322299a-dshm" (OuterVolumeSpecName: "dshm") pod "2d17f830-972b-405b-9a8b-9d824322299a" (UID: "2d17f830-972b-405b-9a8b-9d824322299a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:33:31.068207 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:33:31.068188    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d17f830-972b-405b-9a8b-9d824322299a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2d17f830-972b-405b-9a8b-9d824322299a" (UID: "2d17f830-972b-405b-9a8b-9d824322299a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:33:31.079042 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:33:31.079020    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d17f830-972b-405b-9a8b-9d824322299a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2d17f830-972b-405b-9a8b-9d824322299a" (UID: "2d17f830-972b-405b-9a8b-9d824322299a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:33:31.090746 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:33:31.090717    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d17f830-972b-405b-9a8b-9d824322299a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2d17f830-972b-405b-9a8b-9d824322299a" (UID: "2d17f830-972b-405b-9a8b-9d824322299a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:33:31.166579 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:33:31.166541    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d17f830-972b-405b-9a8b-9d824322299a-tls-certs\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:33:31.166579 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:33:31.166572    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d17f830-972b-405b-9a8b-9d824322299a-cabundle-cert\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:33:31.166579 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:33:31.166584    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d17f830-972b-405b-9a8b-9d824322299a-model-cache\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:33:31.166810 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:33:31.166595    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d17f830-972b-405b-9a8b-9d824322299a-kserve-provision-location\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:33:31.166810 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:33:31.166606    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d17f830-972b-405b-9a8b-9d824322299a-dshm\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:33:31.166810 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:33:31.166616    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d17f830-972b-405b-9a8b-9d824322299a-tmp-dir\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:33:31.166810 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:33:31.166628    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rs45j\" (UniqueName: \"kubernetes.io/projected/2d17f830-972b-405b-9a8b-9d824322299a-kube-api-access-rs45j\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:33:31.166810 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:33:31.166638    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d17f830-972b-405b-9a8b-9d824322299a-home\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:33:31.556371 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:33:31.556331    2575 generic.go:356] "Generic (PLEG): container finished" podID="2d17f830-972b-405b-9a8b-9d824322299a" containerID="a50629a2386e1a5b3532c0b249e84205a057117fb709ccd3983c50992a9e761e" exitCode=0
Sep 17 21:33:31.556779 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:33:31.556387    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c87b9cbcd-fl5nx"
Sep 17 21:33:31.556779 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:33:31.556386    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c87b9cbcd-fl5nx" event={"ID":"2d17f830-972b-405b-9a8b-9d824322299a","Type":"ContainerDied","Data":"a50629a2386e1a5b3532c0b249e84205a057117fb709ccd3983c50992a9e761e"}
Sep 17 21:33:31.556779 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:33:31.556441    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c87b9cbcd-fl5nx" event={"ID":"2d17f830-972b-405b-9a8b-9d824322299a","Type":"ContainerDied","Data":"c7dcb1d0493b22f3451882131180b92c3356e36931a07284fe80fb25780cc33d"}
Sep 17 21:33:31.556779 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:33:31.556470    2575 scope.go:117] "RemoveContainer" containerID="a50629a2386e1a5b3532c0b249e84205a057117fb709ccd3983c50992a9e761e"
Sep 17 21:33:31.565307 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:33:31.565288    2575 scope.go:117] "RemoveContainer" containerID="18f07eb53a06ce920c6fb3890cb7df2a828c6e1d78562e400dd88b15add677f0"
Sep 17 21:33:31.572808 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:33:31.572793    2575 scope.go:117] "RemoveContainer" containerID="a50629a2386e1a5b3532c0b249e84205a057117fb709ccd3983c50992a9e761e"
Sep 17 21:33:31.573102 ip-10-0-139-36 kubenswrapper[2575]: E0917 21:33:31.573081    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a50629a2386e1a5b3532c0b249e84205a057117fb709ccd3983c50992a9e761e\": container with ID starting with a50629a2386e1a5b3532c0b249e84205a057117fb709ccd3983c50992a9e761e not found: ID does not exist" containerID="a50629a2386e1a5b3532c0b249e84205a057117fb709ccd3983c50992a9e761e"
Sep 17 21:33:31.573153 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:33:31.573112    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a50629a2386e1a5b3532c0b249e84205a057117fb709ccd3983c50992a9e761e"} err="failed to get container status \"a50629a2386e1a5b3532c0b249e84205a057117fb709ccd3983c50992a9e761e\": rpc error: code = NotFound desc = could not find container \"a50629a2386e1a5b3532c0b249e84205a057117fb709ccd3983c50992a9e761e\": container with ID starting with a50629a2386e1a5b3532c0b249e84205a057117fb709ccd3983c50992a9e761e not found: ID does not exist"
Sep 17 21:33:31.573153 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:33:31.573129    2575 scope.go:117] "RemoveContainer" containerID="18f07eb53a06ce920c6fb3890cb7df2a828c6e1d78562e400dd88b15add677f0"
Sep 17 21:33:31.573369 ip-10-0-139-36 kubenswrapper[2575]: E0917 21:33:31.573352    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18f07eb53a06ce920c6fb3890cb7df2a828c6e1d78562e400dd88b15add677f0\": container with ID starting with 18f07eb53a06ce920c6fb3890cb7df2a828c6e1d78562e400dd88b15add677f0 not found: ID does not exist" containerID="18f07eb53a06ce920c6fb3890cb7df2a828c6e1d78562e400dd88b15add677f0"
Sep 17 21:33:31.573419 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:33:31.573378    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18f07eb53a06ce920c6fb3890cb7df2a828c6e1d78562e400dd88b15add677f0"} err="failed to get container status \"18f07eb53a06ce920c6fb3890cb7df2a828c6e1d78562e400dd88b15add677f0\": rpc error: code = NotFound desc = could not find container \"18f07eb53a06ce920c6fb3890cb7df2a828c6e1d78562e400dd88b15add677f0\": container with ID starting with 18f07eb53a06ce920c6fb3890cb7df2a828c6e1d78562e400dd88b15add677f0 not found: ID does not exist"
Sep 17 21:33:31.576941 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:33:31.576918    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c87b9cbcd-fl5nx"]
Sep 17 21:33:31.580894 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:33:31.580874    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c87b9cbcd-fl5nx"]
Sep 17 21:33:33.356733 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:33:33.356686    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d17f830-972b-405b-9a8b-9d824322299a" path="/var/lib/kubelet/pods/2d17f830-972b-405b-9a8b-9d824322299a/volumes"
Sep 17 21:33:43.699471 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:33:43.699434    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:33:43.869691 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:33:43.869651    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:33:43.976333 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:33:43.976257    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:34:05.153869 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:34:05.153805    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:34:05.387257 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:34:05.387227    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:34:10.119925 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:34:10.119886    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:34:13.888270 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:34:13.888236    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs"]
Sep 17 21:34:13.888724 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:34:13.888639    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d17f830-972b-405b-9a8b-9d824322299a" containerName="main"
Sep 17 21:34:13.888724 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:34:13.888651    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d17f830-972b-405b-9a8b-9d824322299a" containerName="main"
Sep 17 21:34:13.888724 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:34:13.888668    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d17f830-972b-405b-9a8b-9d824322299a" containerName="storage-initializer"
Sep 17 21:34:13.888724 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:34:13.888674    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d17f830-972b-405b-9a8b-9d824322299a" containerName="storage-initializer"
Sep 17 21:34:13.888890 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:34:13.888748    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d17f830-972b-405b-9a8b-9d824322299a" containerName="main"
Sep 17 21:34:13.891722 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:34:13.891699    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs"
Sep 17 21:34:13.894205 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:34:13.894177    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\""
Sep 17 21:34:13.894529 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:34:13.894510    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 17 21:34:13.895538 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:34:13.895520    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 21:34:13.895646 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:34:13.895558    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 17 21:34:13.896016 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:34:13.896000    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-wmnwr\""
Sep 17 21:34:13.896122 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:34:13.896018    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 17 21:34:13.909380 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:34:13.909354    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs"]
Sep 17 21:34:13.925899 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:34:13.925843    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c7a87eb-f642-4b8e-b684-def699760f05-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs\" (UID: \"1c7a87eb-f642-4b8e-b684-def699760f05\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs"
Sep 17 21:34:13.926045 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:34:13.925903    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1c7a87eb-f642-4b8e-b684-def699760f05-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs\" (UID: \"1c7a87eb-f642-4b8e-b684-def699760f05\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs"
Sep 17 21:34:13.926045 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:34:13.925957    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c7a87eb-f642-4b8e-b684-def699760f05-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs\" (UID: \"1c7a87eb-f642-4b8e-b684-def699760f05\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs"
Sep 17 21:34:13.926045 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:34:13.926034    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c7a87eb-f642-4b8e-b684-def699760f05-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs\" (UID: \"1c7a87eb-f642-4b8e-b684-def699760f05\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs"
Sep 17 21:34:13.926154 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:34:13.926054    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c7a87eb-f642-4b8e-b684-def699760f05-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs\" (UID: \"1c7a87eb-f642-4b8e-b684-def699760f05\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs"
Sep 17 21:34:13.926154 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:34:13.926095    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c7a87eb-f642-4b8e-b684-def699760f05-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs\" (UID: \"1c7a87eb-f642-4b8e-b684-def699760f05\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs"
Sep 17 21:34:13.926154 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:34:13.926124    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c7a87eb-f642-4b8e-b684-def699760f05-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs\" (UID: \"1c7a87eb-f642-4b8e-b684-def699760f05\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs"
Sep 17 21:34:13.926154 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:34:13.926142    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2zvx\" (UniqueName: \"kubernetes.io/projected/1c7a87eb-f642-4b8e-b684-def699760f05-kube-api-access-g2zvx\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs\" (UID: \"1c7a87eb-f642-4b8e-b684-def699760f05\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs"
Sep 17 21:34:13.929318 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:34:13.929290    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:34:14.027365 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:34:14.027329    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2zvx\" (UniqueName: \"kubernetes.io/projected/1c7a87eb-f642-4b8e-b684-def699760f05-kube-api-access-g2zvx\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs\" (UID: \"1c7a87eb-f642-4b8e-b684-def699760f05\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs"
Sep 17 21:34:14.027538 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:34:14.027423    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c7a87eb-f642-4b8e-b684-def699760f05-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs\" (UID: \"1c7a87eb-f642-4b8e-b684-def699760f05\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs"
Sep 17 21:34:14.027538 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:34:14.027442    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1c7a87eb-f642-4b8e-b684-def699760f05-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs\" (UID: \"1c7a87eb-f642-4b8e-b684-def699760f05\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs"
Sep 17 21:34:14.027538 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:34:14.027461    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c7a87eb-f642-4b8e-b684-def699760f05-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs\" (UID: \"1c7a87eb-f642-4b8e-b684-def699760f05\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs"
Sep 17 21:34:14.027538 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:34:14.027519    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c7a87eb-f642-4b8e-b684-def699760f05-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs\" (UID: \"1c7a87eb-f642-4b8e-b684-def699760f05\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs"
Sep 17 21:34:14.027732 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:34:14.027540    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c7a87eb-f642-4b8e-b684-def699760f05-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs\" (UID: \"1c7a87eb-f642-4b8e-b684-def699760f05\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs"
Sep 17 21:34:14.027732 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:34:14.027574    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c7a87eb-f642-4b8e-b684-def699760f05-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs\" (UID: \"1c7a87eb-f642-4b8e-b684-def699760f05\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs"
Sep 17 21:34:14.027732 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:34:14.027611    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c7a87eb-f642-4b8e-b684-def699760f05-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs\" (UID: \"1c7a87eb-f642-4b8e-b684-def699760f05\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs"
Sep 17 21:34:14.027942 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:34:14.027919    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c7a87eb-f642-4b8e-b684-def699760f05-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs\" (UID: \"1c7a87eb-f642-4b8e-b684-def699760f05\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs"
Sep 17 21:34:14.028063 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:34:14.028029    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c7a87eb-f642-4b8e-b684-def699760f05-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs\" (UID: \"1c7a87eb-f642-4b8e-b684-def699760f05\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs"
Sep 17 21:34:14.028149 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:34:14.028056    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c7a87eb-f642-4b8e-b684-def699760f05-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs\" (UID: \"1c7a87eb-f642-4b8e-b684-def699760f05\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs"
Sep 17 21:34:14.028149 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:34:14.028073    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c7a87eb-f642-4b8e-b684-def699760f05-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs\" (UID: \"1c7a87eb-f642-4b8e-b684-def699760f05\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs"
Sep 17 21:34:14.028415 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:34:14.028395    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1c7a87eb-f642-4b8e-b684-def699760f05-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs\" (UID: \"1c7a87eb-f642-4b8e-b684-def699760f05\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs"
Sep 17 21:34:14.029958 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:34:14.029937    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c7a87eb-f642-4b8e-b684-def699760f05-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs\" (UID: \"1c7a87eb-f642-4b8e-b684-def699760f05\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs"
Sep 17 21:34:14.030070 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:34:14.030054    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c7a87eb-f642-4b8e-b684-def699760f05-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs\" (UID: \"1c7a87eb-f642-4b8e-b684-def699760f05\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs"
Sep 17 21:34:14.036226 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:34:14.036206    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2zvx\" (UniqueName: \"kubernetes.io/projected/1c7a87eb-f642-4b8e-b684-def699760f05-kube-api-access-g2zvx\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs\" (UID: \"1c7a87eb-f642-4b8e-b684-def699760f05\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs"
Sep 17 21:34:14.174783 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:34:14.174663    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:34:14.203598 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:34:14.203564    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs"
Sep 17 21:34:14.505594 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:34:14.505562    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs"]
Sep 17 21:34:14.506239 ip-10-0-139-36 kubenswrapper[2575]: W0917 21:34:14.506208    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c7a87eb_f642_4b8e_b684_def699760f05.slice/crio-7c901ff895d214e197154a1f90b7a65d65497ab155a0f414ff6f5496809a6dd0 WatchSource:0}: Error finding container 7c901ff895d214e197154a1f90b7a65d65497ab155a0f414ff6f5496809a6dd0: Status 404 returned error can't find the container with id 7c901ff895d214e197154a1f90b7a65d65497ab155a0f414ff6f5496809a6dd0
Sep 17 21:34:14.769843 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:34:14.769762    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs" event={"ID":"1c7a87eb-f642-4b8e-b684-def699760f05","Type":"ContainerStarted","Data":"e3a4b7743b362b6fa37ed3bb152544437640c5dfbb5f1e2dcfc89b4bcdc4b3dd"}
Sep 17 21:34:14.769843 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:34:14.769816    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs" event={"ID":"1c7a87eb-f642-4b8e-b684-def699760f05","Type":"ContainerStarted","Data":"7c901ff895d214e197154a1f90b7a65d65497ab155a0f414ff6f5496809a6dd0"}
Sep 17 21:34:17.786182 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:34:17.786144    2575 generic.go:356] "Generic (PLEG): container finished" podID="1c7a87eb-f642-4b8e-b684-def699760f05" containerID="e3a4b7743b362b6fa37ed3bb152544437640c5dfbb5f1e2dcfc89b4bcdc4b3dd" exitCode=0
Sep 17 21:34:17.786182 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:34:17.786186    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs" event={"ID":"1c7a87eb-f642-4b8e-b684-def699760f05","Type":"ContainerDied","Data":"e3a4b7743b362b6fa37ed3bb152544437640c5dfbb5f1e2dcfc89b4bcdc4b3dd"}
Sep 17 21:34:18.794374 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:34:18.794339    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs" event={"ID":"1c7a87eb-f642-4b8e-b684-def699760f05","Type":"ContainerStarted","Data":"ae554a6f290b8854634c6815f9c0fdf332614471ea57eafa64eb3f8b84f16df2"}
Sep 17 21:34:18.832418 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:34:18.832356    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs" podStartSLOduration=3.587490381 podStartE2EDuration="5.832335775s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.244845394s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:34:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:34:18.830432994 +0000 UTC m=+2723.982877369" watchObservedRunningTime="2026-09-17 21:34:18.832335775 +0000 UTC m=+2723.984780151"
Sep 17 21:34:24.203900 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:34:24.203863    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs"
Sep 17 21:34:24.204288 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:34:24.203946    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs"
Sep 17 21:34:24.205412 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:34:24.205383    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs" podUID="1c7a87eb-f642-4b8e-b684-def699760f05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Sep 17 21:34:29.379828 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:34:29.379787    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:34:33.225645 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:34:33.225607    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:34:33.445569 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:34:33.445526    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:34:33.667285 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:34:33.667192    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:34:34.204308 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:34:34.204269    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs" podUID="1c7a87eb-f642-4b8e-b684-def699760f05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Sep 17 21:34:34.274574 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:34:34.274540    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:34:40.993101 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:34:40.993061    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:34:41.363383 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:34:41.363297    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:34:41.543888 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:34:41.543828    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:34:41.826394 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:34:41.826356    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:34:44.204652 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:34:44.204604    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs" podUID="1c7a87eb-f642-4b8e-b684-def699760f05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Sep 17 21:34:47.016275 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:34:47.016236    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:34:47.129150 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:34:47.129103    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:34:48.961488 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:34:48.961441    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:34:49.059303 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:34:49.059267    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:34:54.204772 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:34:54.204728    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs" podUID="1c7a87eb-f642-4b8e-b684-def699760f05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Sep 17 21:35:00.019911 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:35:00.019863    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:35:04.204519 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:35:04.204467    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs" podUID="1c7a87eb-f642-4b8e-b684-def699760f05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Sep 17 21:35:04.421944 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:35:04.421906    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:35:06.501234 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:35:06.501192    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:35:06.599616 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:35:06.599565    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:35:13.414615 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:35:13.414574    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:35:13.484274 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:35:13.484234    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:35:14.204636 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:35:14.204585    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs" podUID="1c7a87eb-f642-4b8e-b684-def699760f05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Sep 17 21:35:24.204916 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:35:24.204793    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs" podUID="1c7a87eb-f642-4b8e-b684-def699760f05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Sep 17 21:35:34.204029 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:35:34.203991    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs" podUID="1c7a87eb-f642-4b8e-b684-def699760f05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Sep 17 21:35:44.204731 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:35:44.204667    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs" podUID="1c7a87eb-f642-4b8e-b684-def699760f05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Sep 17 21:35:54.204152 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:35:54.204095    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs" podUID="1c7a87eb-f642-4b8e-b684-def699760f05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Sep 17 21:36:04.204602 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:36:04.204552    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs" podUID="1c7a87eb-f642-4b8e-b684-def699760f05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Sep 17 21:36:14.214154 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:36:14.214120    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs"
Sep 17 21:36:14.222387 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:36:14.222356    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs"
Sep 17 21:36:23.145130 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:36:23.145094    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs"]
Sep 17 21:36:23.156421 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:36:23.156373    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:36:23.235772 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:36:23.235740    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:36:31.733009 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:36:31.732973    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:36:31.791267 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:36:31.791235    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:36:38.169668 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:36:38.169601    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs" podUID="1c7a87eb-f642-4b8e-b684-def699760f05" containerName="main" containerID="cri-o://ae554a6f290b8854634c6815f9c0fdf332614471ea57eafa64eb3f8b84f16df2" gracePeriod=45
Sep 17 21:36:38.845756 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:36:38.845729    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs"
Sep 17 21:36:38.857047 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:36:38.857025    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1c7a87eb-f642-4b8e-b684-def699760f05-cabundle-cert\") pod \"1c7a87eb-f642-4b8e-b684-def699760f05\" (UID: \"1c7a87eb-f642-4b8e-b684-def699760f05\") "
Sep 17 21:36:38.857110 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:36:38.857060    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c7a87eb-f642-4b8e-b684-def699760f05-kserve-provision-location\") pod \"1c7a87eb-f642-4b8e-b684-def699760f05\" (UID: \"1c7a87eb-f642-4b8e-b684-def699760f05\") "
Sep 17 21:36:38.857110 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:36:38.857082    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g2zvx\" (UniqueName: \"kubernetes.io/projected/1c7a87eb-f642-4b8e-b684-def699760f05-kube-api-access-g2zvx\") pod \"1c7a87eb-f642-4b8e-b684-def699760f05\" (UID: \"1c7a87eb-f642-4b8e-b684-def699760f05\") "
Sep 17 21:36:38.857227 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:36:38.857206    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c7a87eb-f642-4b8e-b684-def699760f05-model-cache\") pod \"1c7a87eb-f642-4b8e-b684-def699760f05\" (UID: \"1c7a87eb-f642-4b8e-b684-def699760f05\") "
Sep 17 21:36:38.857284 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:36:38.857269    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c7a87eb-f642-4b8e-b684-def699760f05-dshm\") pod \"1c7a87eb-f642-4b8e-b684-def699760f05\" (UID: \"1c7a87eb-f642-4b8e-b684-def699760f05\") "
Sep 17 21:36:38.857349 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:36:38.857334    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c7a87eb-f642-4b8e-b684-def699760f05-tmp-dir\") pod \"1c7a87eb-f642-4b8e-b684-def699760f05\" (UID: \"1c7a87eb-f642-4b8e-b684-def699760f05\") "
Sep 17 21:36:38.857407 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:36:38.857369    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c7a87eb-f642-4b8e-b684-def699760f05-home\") pod \"1c7a87eb-f642-4b8e-b684-def699760f05\" (UID: \"1c7a87eb-f642-4b8e-b684-def699760f05\") "
Sep 17 21:36:38.857463 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:36:38.857420    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c7a87eb-f642-4b8e-b684-def699760f05-tls-certs\") pod \"1c7a87eb-f642-4b8e-b684-def699760f05\" (UID: \"1c7a87eb-f642-4b8e-b684-def699760f05\") "
Sep 17 21:36:38.857520 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:36:38.857489    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c7a87eb-f642-4b8e-b684-def699760f05-model-cache" (OuterVolumeSpecName: "model-cache") pod "1c7a87eb-f642-4b8e-b684-def699760f05" (UID: "1c7a87eb-f642-4b8e-b684-def699760f05"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:36:38.857585 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:36:38.857513    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c7a87eb-f642-4b8e-b684-def699760f05-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1c7a87eb-f642-4b8e-b684-def699760f05" (UID: "1c7a87eb-f642-4b8e-b684-def699760f05"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:36:38.857749 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:36:38.857729    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1c7a87eb-f642-4b8e-b684-def699760f05-cabundle-cert\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:36:38.857870 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:36:38.857753    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c7a87eb-f642-4b8e-b684-def699760f05-model-cache\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:36:38.858231 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:36:38.858200    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c7a87eb-f642-4b8e-b684-def699760f05-home" (OuterVolumeSpecName: "home") pod "1c7a87eb-f642-4b8e-b684-def699760f05" (UID: "1c7a87eb-f642-4b8e-b684-def699760f05"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:36:38.859535 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:36:38.859512    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c7a87eb-f642-4b8e-b684-def699760f05-kube-api-access-g2zvx" (OuterVolumeSpecName: "kube-api-access-g2zvx") pod "1c7a87eb-f642-4b8e-b684-def699760f05" (UID: "1c7a87eb-f642-4b8e-b684-def699760f05"). InnerVolumeSpecName "kube-api-access-g2zvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:36:38.859638 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:36:38.859535    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c7a87eb-f642-4b8e-b684-def699760f05-dshm" (OuterVolumeSpecName: "dshm") pod "1c7a87eb-f642-4b8e-b684-def699760f05" (UID: "1c7a87eb-f642-4b8e-b684-def699760f05"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:36:38.859821 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:36:38.859803    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c7a87eb-f642-4b8e-b684-def699760f05-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1c7a87eb-f642-4b8e-b684-def699760f05" (UID: "1c7a87eb-f642-4b8e-b684-def699760f05"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:36:38.869491 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:36:38.869466    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c7a87eb-f642-4b8e-b684-def699760f05-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1c7a87eb-f642-4b8e-b684-def699760f05" (UID: "1c7a87eb-f642-4b8e-b684-def699760f05"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:36:38.887208 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:36:38.887174    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c7a87eb-f642-4b8e-b684-def699760f05-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1c7a87eb-f642-4b8e-b684-def699760f05" (UID: "1c7a87eb-f642-4b8e-b684-def699760f05"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:36:38.958204 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:36:38.958175    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c7a87eb-f642-4b8e-b684-def699760f05-tls-certs\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:36:38.958204 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:36:38.958202    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c7a87eb-f642-4b8e-b684-def699760f05-kserve-provision-location\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:36:38.958381 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:36:38.958212    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2zvx\" (UniqueName: \"kubernetes.io/projected/1c7a87eb-f642-4b8e-b684-def699760f05-kube-api-access-g2zvx\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:36:38.958381 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:36:38.958220    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c7a87eb-f642-4b8e-b684-def699760f05-dshm\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:36:38.958381 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:36:38.958228    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c7a87eb-f642-4b8e-b684-def699760f05-tmp-dir\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:36:38.958381 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:36:38.958236    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c7a87eb-f642-4b8e-b684-def699760f05-home\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:36:39.526689 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:36:39.526651    2575 generic.go:356] "Generic (PLEG): container finished" podID="1c7a87eb-f642-4b8e-b684-def699760f05" containerID="ae554a6f290b8854634c6815f9c0fdf332614471ea57eafa64eb3f8b84f16df2" exitCode=0
Sep 17 21:36:39.527134 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:36:39.526704    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs"
Sep 17 21:36:39.527134 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:36:39.526744    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs" event={"ID":"1c7a87eb-f642-4b8e-b684-def699760f05","Type":"ContainerDied","Data":"ae554a6f290b8854634c6815f9c0fdf332614471ea57eafa64eb3f8b84f16df2"}
Sep 17 21:36:39.527134 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:36:39.526809    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs" event={"ID":"1c7a87eb-f642-4b8e-b684-def699760f05","Type":"ContainerDied","Data":"7c901ff895d214e197154a1f90b7a65d65497ab155a0f414ff6f5496809a6dd0"}
Sep 17 21:36:39.527134 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:36:39.526835    2575 scope.go:117] "RemoveContainer" containerID="ae554a6f290b8854634c6815f9c0fdf332614471ea57eafa64eb3f8b84f16df2"
Sep 17 21:36:39.535649 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:36:39.535626    2575 scope.go:117] "RemoveContainer" containerID="e3a4b7743b362b6fa37ed3bb152544437640c5dfbb5f1e2dcfc89b4bcdc4b3dd"
Sep 17 21:36:39.543369 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:36:39.543350    2575 scope.go:117] "RemoveContainer" containerID="ae554a6f290b8854634c6815f9c0fdf332614471ea57eafa64eb3f8b84f16df2"
Sep 17 21:36:39.543631 ip-10-0-139-36 kubenswrapper[2575]: E0917 21:36:39.543613    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae554a6f290b8854634c6815f9c0fdf332614471ea57eafa64eb3f8b84f16df2\": container with ID starting with ae554a6f290b8854634c6815f9c0fdf332614471ea57eafa64eb3f8b84f16df2 not found: ID does not exist" containerID="ae554a6f290b8854634c6815f9c0fdf332614471ea57eafa64eb3f8b84f16df2"
Sep 17 21:36:39.543682 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:36:39.543640    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae554a6f290b8854634c6815f9c0fdf332614471ea57eafa64eb3f8b84f16df2"} err="failed to get container status \"ae554a6f290b8854634c6815f9c0fdf332614471ea57eafa64eb3f8b84f16df2\": rpc error: code = NotFound desc = could not find container \"ae554a6f290b8854634c6815f9c0fdf332614471ea57eafa64eb3f8b84f16df2\": container with ID starting with ae554a6f290b8854634c6815f9c0fdf332614471ea57eafa64eb3f8b84f16df2 not found: ID does not exist"
Sep 17 21:36:39.543682 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:36:39.543656    2575 scope.go:117] "RemoveContainer" containerID="e3a4b7743b362b6fa37ed3bb152544437640c5dfbb5f1e2dcfc89b4bcdc4b3dd"
Sep 17 21:36:39.543912 ip-10-0-139-36 kubenswrapper[2575]: E0917 21:36:39.543891    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3a4b7743b362b6fa37ed3bb152544437640c5dfbb5f1e2dcfc89b4bcdc4b3dd\": container with ID starting with e3a4b7743b362b6fa37ed3bb152544437640c5dfbb5f1e2dcfc89b4bcdc4b3dd not found: ID does not exist" containerID="e3a4b7743b362b6fa37ed3bb152544437640c5dfbb5f1e2dcfc89b4bcdc4b3dd"
Sep 17 21:36:39.543983 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:36:39.543924    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3a4b7743b362b6fa37ed3bb152544437640c5dfbb5f1e2dcfc89b4bcdc4b3dd"} err="failed to get container status \"e3a4b7743b362b6fa37ed3bb152544437640c5dfbb5f1e2dcfc89b4bcdc4b3dd\": rpc error: code = NotFound desc = could not find container \"e3a4b7743b362b6fa37ed3bb152544437640c5dfbb5f1e2dcfc89b4bcdc4b3dd\": container with ID starting with e3a4b7743b362b6fa37ed3bb152544437640c5dfbb5f1e2dcfc89b4bcdc4b3dd not found: ID does not exist"
Sep 17 21:36:39.550657 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:36:39.550632    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs"]
Sep 17 21:36:39.557972 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:36:39.557948    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-cfh5sqs"]
Sep 17 21:36:41.357181 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:36:41.357144    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c7a87eb-f642-4b8e-b684-def699760f05" path="/var/lib/kubelet/pods/1c7a87eb-f642-4b8e-b684-def699760f05/volumes"
Sep 17 21:36:43.764051 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:36:43.764013    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:36:43.893812 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:36:43.893766    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:37:00.315018 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:37:00.314980    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-8mj66"]
Sep 17 21:37:00.315439 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:37:00.315344    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c7a87eb-f642-4b8e-b684-def699760f05" containerName="storage-initializer"
Sep 17 21:37:00.315439 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:37:00.315356    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c7a87eb-f642-4b8e-b684-def699760f05" containerName="storage-initializer"
Sep 17 21:37:00.315439 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:37:00.315374    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c7a87eb-f642-4b8e-b684-def699760f05" containerName="main"
Sep 17 21:37:00.315439 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:37:00.315379    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c7a87eb-f642-4b8e-b684-def699760f05" containerName="main"
Sep 17 21:37:00.315439 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:37:00.315439    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c7a87eb-f642-4b8e-b684-def699760f05" containerName="main"
Sep 17 21:37:00.320330 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:37:00.320314    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-8mj66"
Sep 17 21:37:00.325184 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:37:00.325158    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 17 21:37:00.325314 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:37:00.325203    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 17 21:37:00.325862 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:37:00.325820    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 17 21:37:00.325977 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:37:00.325894    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Sep 17 21:37:00.325977 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:37:00.325915    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 21:37:00.325977 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:37:00.325894    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-q56p2\""
Sep 17 21:37:00.354785 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:37:00.354764    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-8mj66"]
Sep 17 21:37:00.426823 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:37:00.426786    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d81b305-7eb2-4ce4-96e0-08476ca9b9e8-model-cache\") pod \"stop-feature-test-kserve-555fc44b76-8mj66\" (UID: \"7d81b305-7eb2-4ce4-96e0-08476ca9b9e8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-8mj66"
Sep 17 21:37:00.427030 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:37:00.426830    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d81b305-7eb2-4ce4-96e0-08476ca9b9e8-tmp-dir\") pod \"stop-feature-test-kserve-555fc44b76-8mj66\" (UID: \"7d81b305-7eb2-4ce4-96e0-08476ca9b9e8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-8mj66"
Sep 17 21:37:00.427030 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:37:00.426869    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d81b305-7eb2-4ce4-96e0-08476ca9b9e8-cabundle-cert\") pod \"stop-feature-test-kserve-555fc44b76-8mj66\" (UID: \"7d81b305-7eb2-4ce4-96e0-08476ca9b9e8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-8mj66"
Sep 17 21:37:00.427030 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:37:00.426896    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d81b305-7eb2-4ce4-96e0-08476ca9b9e8-dshm\") pod \"stop-feature-test-kserve-555fc44b76-8mj66\" (UID: \"7d81b305-7eb2-4ce4-96e0-08476ca9b9e8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-8mj66"
Sep 17 21:37:00.427030 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:37:00.426931    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d81b305-7eb2-4ce4-96e0-08476ca9b9e8-home\") pod \"stop-feature-test-kserve-555fc44b76-8mj66\" (UID: \"7d81b305-7eb2-4ce4-96e0-08476ca9b9e8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-8mj66"
Sep 17 21:37:00.427030 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:37:00.426992    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz4nv\" (UniqueName: \"kubernetes.io/projected/7d81b305-7eb2-4ce4-96e0-08476ca9b9e8-kube-api-access-hz4nv\") pod \"stop-feature-test-kserve-555fc44b76-8mj66\" (UID: \"7d81b305-7eb2-4ce4-96e0-08476ca9b9e8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-8mj66"
Sep 17 21:37:00.427282 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:37:00.427107    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d81b305-7eb2-4ce4-96e0-08476ca9b9e8-kserve-provision-location\") pod \"stop-feature-test-kserve-555fc44b76-8mj66\" (UID: \"7d81b305-7eb2-4ce4-96e0-08476ca9b9e8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-8mj66"
Sep 17 21:37:00.427282 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:37:00.427146    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d81b305-7eb2-4ce4-96e0-08476ca9b9e8-tls-certs\") pod \"stop-feature-test-kserve-555fc44b76-8mj66\" (UID: \"7d81b305-7eb2-4ce4-96e0-08476ca9b9e8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-8mj66"
Sep 17 21:37:00.470689 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:37:00.470650    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:37:00.527854 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:37:00.527822    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz4nv\" (UniqueName: \"kubernetes.io/projected/7d81b305-7eb2-4ce4-96e0-08476ca9b9e8-kube-api-access-hz4nv\") pod \"stop-feature-test-kserve-555fc44b76-8mj66\" (UID: \"7d81b305-7eb2-4ce4-96e0-08476ca9b9e8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-8mj66"
Sep 17 21:37:00.528019 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:37:00.527916    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d81b305-7eb2-4ce4-96e0-08476ca9b9e8-kserve-provision-location\") pod \"stop-feature-test-kserve-555fc44b76-8mj66\" (UID: \"7d81b305-7eb2-4ce4-96e0-08476ca9b9e8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-8mj66"
Sep 17 21:37:00.528019 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:37:00.527940    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d81b305-7eb2-4ce4-96e0-08476ca9b9e8-tls-certs\") pod \"stop-feature-test-kserve-555fc44b76-8mj66\" (UID: \"7d81b305-7eb2-4ce4-96e0-08476ca9b9e8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-8mj66"
Sep 17 21:37:00.528019 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:37:00.528005    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d81b305-7eb2-4ce4-96e0-08476ca9b9e8-model-cache\") pod \"stop-feature-test-kserve-555fc44b76-8mj66\" (UID: \"7d81b305-7eb2-4ce4-96e0-08476ca9b9e8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-8mj66"
Sep 17 21:37:00.528190 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:37:00.528024    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d81b305-7eb2-4ce4-96e0-08476ca9b9e8-tmp-dir\") pod \"stop-feature-test-kserve-555fc44b76-8mj66\" (UID: \"7d81b305-7eb2-4ce4-96e0-08476ca9b9e8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-8mj66"
Sep 17 21:37:00.528190 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:37:00.528038    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d81b305-7eb2-4ce4-96e0-08476ca9b9e8-cabundle-cert\") pod \"stop-feature-test-kserve-555fc44b76-8mj66\" (UID: \"7d81b305-7eb2-4ce4-96e0-08476ca9b9e8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-8mj66"
Sep 17 21:37:00.528190 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:37:00.528065    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d81b305-7eb2-4ce4-96e0-08476ca9b9e8-dshm\") pod \"stop-feature-test-kserve-555fc44b76-8mj66\" (UID: \"7d81b305-7eb2-4ce4-96e0-08476ca9b9e8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-8mj66"
Sep 17 21:37:00.528190 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:37:00.528099    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d81b305-7eb2-4ce4-96e0-08476ca9b9e8-home\") pod \"stop-feature-test-kserve-555fc44b76-8mj66\" (UID: \"7d81b305-7eb2-4ce4-96e0-08476ca9b9e8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-8mj66"
Sep 17 21:37:00.528399 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:37:00.528374    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d81b305-7eb2-4ce4-96e0-08476ca9b9e8-kserve-provision-location\") pod \"stop-feature-test-kserve-555fc44b76-8mj66\" (UID: \"7d81b305-7eb2-4ce4-96e0-08476ca9b9e8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-8mj66"
Sep 17 21:37:00.528462 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:37:00.528421    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d81b305-7eb2-4ce4-96e0-08476ca9b9e8-model-cache\") pod \"stop-feature-test-kserve-555fc44b76-8mj66\" (UID: \"7d81b305-7eb2-4ce4-96e0-08476ca9b9e8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-8mj66"
Sep 17 21:37:00.528517 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:37:00.528472    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d81b305-7eb2-4ce4-96e0-08476ca9b9e8-tmp-dir\") pod \"stop-feature-test-kserve-555fc44b76-8mj66\" (UID: \"7d81b305-7eb2-4ce4-96e0-08476ca9b9e8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-8mj66"
Sep 17 21:37:00.528557 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:37:00.528530    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d81b305-7eb2-4ce4-96e0-08476ca9b9e8-home\") pod \"stop-feature-test-kserve-555fc44b76-8mj66\" (UID: \"7d81b305-7eb2-4ce4-96e0-08476ca9b9e8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-8mj66"
Sep 17 21:37:00.528803 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:37:00.528778    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d81b305-7eb2-4ce4-96e0-08476ca9b9e8-cabundle-cert\") pod \"stop-feature-test-kserve-555fc44b76-8mj66\" (UID: \"7d81b305-7eb2-4ce4-96e0-08476ca9b9e8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-8mj66"
Sep 17 21:37:00.530543 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:37:00.530517    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d81b305-7eb2-4ce4-96e0-08476ca9b9e8-dshm\") pod \"stop-feature-test-kserve-555fc44b76-8mj66\" (UID: \"7d81b305-7eb2-4ce4-96e0-08476ca9b9e8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-8mj66"
Sep 17 21:37:00.530758 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:37:00.530743    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d81b305-7eb2-4ce4-96e0-08476ca9b9e8-tls-certs\") pod \"stop-feature-test-kserve-555fc44b76-8mj66\" (UID: \"7d81b305-7eb2-4ce4-96e0-08476ca9b9e8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-8mj66"
Sep 17 21:37:00.535619 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:37:00.535601    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz4nv\" (UniqueName: \"kubernetes.io/projected/7d81b305-7eb2-4ce4-96e0-08476ca9b9e8-kube-api-access-hz4nv\") pod \"stop-feature-test-kserve-555fc44b76-8mj66\" (UID: \"7d81b305-7eb2-4ce4-96e0-08476ca9b9e8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-8mj66"
Sep 17 21:37:00.630648 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:37:00.630565    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-8mj66"
Sep 17 21:37:00.916656 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:37:00.916626    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-8mj66"]
Sep 17 21:37:00.917904 ip-10-0-139-36 kubenswrapper[2575]: W0917 21:37:00.917876    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d81b305_7eb2_4ce4_96e0_08476ca9b9e8.slice/crio-62e1471eca90ed5514227b39c15841c7eede8e00345215894ae5faaa50635e68 WatchSource:0}: Error finding container 62e1471eca90ed5514227b39c15841c7eede8e00345215894ae5faaa50635e68: Status 404 returned error can't find the container with id 62e1471eca90ed5514227b39c15841c7eede8e00345215894ae5faaa50635e68
Sep 17 21:37:01.610670 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:37:01.610628    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:37:01.639500 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:37:01.639466    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-8mj66" event={"ID":"7d81b305-7eb2-4ce4-96e0-08476ca9b9e8","Type":"ContainerStarted","Data":"69c9b76dd85dc41130cc100923dda34065f94d52fa86b7b997d58f022dd7e2e5"}
Sep 17 21:37:01.639677 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:37:01.639535    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-8mj66" event={"ID":"7d81b305-7eb2-4ce4-96e0-08476ca9b9e8","Type":"ContainerStarted","Data":"62e1471eca90ed5514227b39c15841c7eede8e00345215894ae5faaa50635e68"}
Sep 17 21:37:03.652370 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:37:03.652332    2575 generic.go:356] "Generic (PLEG): container finished" podID="7d81b305-7eb2-4ce4-96e0-08476ca9b9e8" containerID="69c9b76dd85dc41130cc100923dda34065f94d52fa86b7b997d58f022dd7e2e5" exitCode=0
Sep 17 21:37:03.652767 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:37:03.652385    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-8mj66" event={"ID":"7d81b305-7eb2-4ce4-96e0-08476ca9b9e8","Type":"ContainerDied","Data":"69c9b76dd85dc41130cc100923dda34065f94d52fa86b7b997d58f022dd7e2e5"}
Sep 17 21:37:04.659088 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:37:04.659047    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-8mj66" event={"ID":"7d81b305-7eb2-4ce4-96e0-08476ca9b9e8","Type":"ContainerStarted","Data":"9d124a8162b884ae2424e7f435b746adc2f24157ca9869ad373429c42a76e055"}
Sep 17 21:37:04.691963 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:37:04.691909    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-8mj66" podStartSLOduration=2.506510075 podStartE2EDuration="4.691893296s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.185383221s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:37:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:37:04.689809116 +0000 UTC m=+2889.842253518" watchObservedRunningTime="2026-09-17 21:37:04.691893296 +0000 UTC m=+2889.844337671"
Sep 17 21:37:05.630894 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:37:05.630860    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-8mj66"
Sep 17 21:37:09.329911 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:37:09.329866    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:37:10.631539 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:37:10.631505    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-8mj66"
Sep 17 21:37:10.632820 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:37:10.632791    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-8mj66" podUID="7d81b305-7eb2-4ce4-96e0-08476ca9b9e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 17 21:37:13.229385 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:37:13.229352    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:37:13.328987 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:37:13.328948    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:37:20.459014 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:37:20.458978    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:37:20.631457 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:37:20.631411    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-8mj66" podUID="7d81b305-7eb2-4ce4-96e0-08476ca9b9e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 17 21:37:24.029659 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:37:24.029621    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:37:24.289920 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:37:24.289728    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:37:27.849841 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:37:27.849805    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:37:27.949473 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:37:27.949431    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:37:30.631862 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:37:30.631800    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-8mj66" podUID="7d81b305-7eb2-4ce4-96e0-08476ca9b9e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 17 21:37:40.631364 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:37:40.631316    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-8mj66" podUID="7d81b305-7eb2-4ce4-96e0-08476ca9b9e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 17 21:37:50.631856 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:37:50.631810    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-8mj66" podUID="7d81b305-7eb2-4ce4-96e0-08476ca9b9e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 17 21:37:55.920654 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:37:55.920606    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:38:00.631999 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:38:00.631955    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-8mj66" podUID="7d81b305-7eb2-4ce4-96e0-08476ca9b9e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 17 21:38:10.631819 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:38:10.631771    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-8mj66" podUID="7d81b305-7eb2-4ce4-96e0-08476ca9b9e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 17 21:38:17.551352 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:38:17.551312    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:38:20.631792 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:38:20.631747    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-8mj66" podUID="7d81b305-7eb2-4ce4-96e0-08476ca9b9e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 17 21:38:30.631971 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:38:30.631931    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-8mj66" podUID="7d81b305-7eb2-4ce4-96e0-08476ca9b9e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 17 21:38:40.631617 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:38:40.631570    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-8mj66" podUID="7d81b305-7eb2-4ce4-96e0-08476ca9b9e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 17 21:38:50.631352 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:38:50.631308    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-8mj66" podUID="7d81b305-7eb2-4ce4-96e0-08476ca9b9e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 17 21:39:00.642074 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:39:00.641998    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-8mj66"
Sep 17 21:39:00.650695 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:39:00.650655    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-8mj66"
Sep 17 21:39:02.822366 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:39:02.822324    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:39:02.825820 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:39:02.825786    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-8mj66"]
Sep 17 21:39:02.914799 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:39:02.914748    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:39:17.849013 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:39:17.848948    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-8mj66" podUID="7d81b305-7eb2-4ce4-96e0-08476ca9b9e8" containerName="main" containerID="cri-o://9d124a8162b884ae2424e7f435b746adc2f24157ca9869ad373429c42a76e055" gracePeriod=45
Sep 17 21:39:18.603483 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:39:18.603459    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-8mj66"
Sep 17 21:39:18.692008 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:39:18.691925    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d81b305-7eb2-4ce4-96e0-08476ca9b9e8-dshm\") pod \"7d81b305-7eb2-4ce4-96e0-08476ca9b9e8\" (UID: \"7d81b305-7eb2-4ce4-96e0-08476ca9b9e8\") "
Sep 17 21:39:18.692008 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:39:18.691982    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d81b305-7eb2-4ce4-96e0-08476ca9b9e8-tmp-dir\") pod \"7d81b305-7eb2-4ce4-96e0-08476ca9b9e8\" (UID: \"7d81b305-7eb2-4ce4-96e0-08476ca9b9e8\") "
Sep 17 21:39:18.692008 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:39:18.692000    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d81b305-7eb2-4ce4-96e0-08476ca9b9e8-kserve-provision-location\") pod \"7d81b305-7eb2-4ce4-96e0-08476ca9b9e8\" (UID: \"7d81b305-7eb2-4ce4-96e0-08476ca9b9e8\") "
Sep 17 21:39:18.692310 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:39:18.692047    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d81b305-7eb2-4ce4-96e0-08476ca9b9e8-tls-certs\") pod \"7d81b305-7eb2-4ce4-96e0-08476ca9b9e8\" (UID: \"7d81b305-7eb2-4ce4-96e0-08476ca9b9e8\") "
Sep 17 21:39:18.692310 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:39:18.692087    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d81b305-7eb2-4ce4-96e0-08476ca9b9e8-model-cache\") pod \"7d81b305-7eb2-4ce4-96e0-08476ca9b9e8\" (UID: \"7d81b305-7eb2-4ce4-96e0-08476ca9b9e8\") "
Sep 17 21:39:18.692310 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:39:18.692232    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hz4nv\" (UniqueName: \"kubernetes.io/projected/7d81b305-7eb2-4ce4-96e0-08476ca9b9e8-kube-api-access-hz4nv\") pod \"7d81b305-7eb2-4ce4-96e0-08476ca9b9e8\" (UID: \"7d81b305-7eb2-4ce4-96e0-08476ca9b9e8\") "
Sep 17 21:39:18.692310 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:39:18.692289    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d81b305-7eb2-4ce4-96e0-08476ca9b9e8-home\") pod \"7d81b305-7eb2-4ce4-96e0-08476ca9b9e8\" (UID: \"7d81b305-7eb2-4ce4-96e0-08476ca9b9e8\") "
Sep 17 21:39:18.692535 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:39:18.692314    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d81b305-7eb2-4ce4-96e0-08476ca9b9e8-model-cache" (OuterVolumeSpecName: "model-cache") pod "7d81b305-7eb2-4ce4-96e0-08476ca9b9e8" (UID: "7d81b305-7eb2-4ce4-96e0-08476ca9b9e8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:39:18.692535 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:39:18.692331    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d81b305-7eb2-4ce4-96e0-08476ca9b9e8-cabundle-cert\") pod \"7d81b305-7eb2-4ce4-96e0-08476ca9b9e8\" (UID: \"7d81b305-7eb2-4ce4-96e0-08476ca9b9e8\") "
Sep 17 21:39:18.692669 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:39:18.692649    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d81b305-7eb2-4ce4-96e0-08476ca9b9e8-model-cache\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:39:18.692984 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:39:18.692956    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d81b305-7eb2-4ce4-96e0-08476ca9b9e8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7d81b305-7eb2-4ce4-96e0-08476ca9b9e8" (UID: "7d81b305-7eb2-4ce4-96e0-08476ca9b9e8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:39:18.693096 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:39:18.693011    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d81b305-7eb2-4ce4-96e0-08476ca9b9e8-home" (OuterVolumeSpecName: "home") pod "7d81b305-7eb2-4ce4-96e0-08476ca9b9e8" (UID: "7d81b305-7eb2-4ce4-96e0-08476ca9b9e8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:39:18.694465 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:39:18.694430    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d81b305-7eb2-4ce4-96e0-08476ca9b9e8-dshm" (OuterVolumeSpecName: "dshm") pod "7d81b305-7eb2-4ce4-96e0-08476ca9b9e8" (UID: "7d81b305-7eb2-4ce4-96e0-08476ca9b9e8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:39:18.694580 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:39:18.694492    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d81b305-7eb2-4ce4-96e0-08476ca9b9e8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7d81b305-7eb2-4ce4-96e0-08476ca9b9e8" (UID: "7d81b305-7eb2-4ce4-96e0-08476ca9b9e8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:39:18.695103 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:39:18.695083    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d81b305-7eb2-4ce4-96e0-08476ca9b9e8-kube-api-access-hz4nv" (OuterVolumeSpecName: "kube-api-access-hz4nv") pod "7d81b305-7eb2-4ce4-96e0-08476ca9b9e8" (UID: "7d81b305-7eb2-4ce4-96e0-08476ca9b9e8"). InnerVolumeSpecName "kube-api-access-hz4nv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:39:18.704931 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:39:18.704904    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d81b305-7eb2-4ce4-96e0-08476ca9b9e8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7d81b305-7eb2-4ce4-96e0-08476ca9b9e8" (UID: "7d81b305-7eb2-4ce4-96e0-08476ca9b9e8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:39:18.716191 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:39:18.716167    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d81b305-7eb2-4ce4-96e0-08476ca9b9e8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7d81b305-7eb2-4ce4-96e0-08476ca9b9e8" (UID: "7d81b305-7eb2-4ce4-96e0-08476ca9b9e8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:39:18.793339 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:39:18.793302    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hz4nv\" (UniqueName: \"kubernetes.io/projected/7d81b305-7eb2-4ce4-96e0-08476ca9b9e8-kube-api-access-hz4nv\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:39:18.793339 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:39:18.793336    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d81b305-7eb2-4ce4-96e0-08476ca9b9e8-home\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:39:18.793339 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:39:18.793346    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d81b305-7eb2-4ce4-96e0-08476ca9b9e8-cabundle-cert\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:39:18.793539 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:39:18.793356    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d81b305-7eb2-4ce4-96e0-08476ca9b9e8-dshm\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:39:18.793539 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:39:18.793364    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d81b305-7eb2-4ce4-96e0-08476ca9b9e8-tmp-dir\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:39:18.793539 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:39:18.793371    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d81b305-7eb2-4ce4-96e0-08476ca9b9e8-kserve-provision-location\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:39:18.793539 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:39:18.793379    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d81b305-7eb2-4ce4-96e0-08476ca9b9e8-tls-certs\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:39:19.361562 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:39:19.361529    2575 generic.go:356] "Generic (PLEG): container finished" podID="7d81b305-7eb2-4ce4-96e0-08476ca9b9e8" containerID="9d124a8162b884ae2424e7f435b746adc2f24157ca9869ad373429c42a76e055" exitCode=0
Sep 17 21:39:19.362069 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:39:19.361596    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-8mj66"
Sep 17 21:39:19.362069 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:39:19.361615    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-8mj66" event={"ID":"7d81b305-7eb2-4ce4-96e0-08476ca9b9e8","Type":"ContainerDied","Data":"9d124a8162b884ae2424e7f435b746adc2f24157ca9869ad373429c42a76e055"}
Sep 17 21:39:19.362069 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:39:19.361687    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-8mj66" event={"ID":"7d81b305-7eb2-4ce4-96e0-08476ca9b9e8","Type":"ContainerDied","Data":"62e1471eca90ed5514227b39c15841c7eede8e00345215894ae5faaa50635e68"}
Sep 17 21:39:19.362069 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:39:19.361717    2575 scope.go:117] "RemoveContainer" containerID="9d124a8162b884ae2424e7f435b746adc2f24157ca9869ad373429c42a76e055"
Sep 17 21:39:19.371666 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:39:19.371650    2575 scope.go:117] "RemoveContainer" containerID="69c9b76dd85dc41130cc100923dda34065f94d52fa86b7b997d58f022dd7e2e5"
Sep 17 21:39:19.379581 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:39:19.379561    2575 scope.go:117] "RemoveContainer" containerID="9d124a8162b884ae2424e7f435b746adc2f24157ca9869ad373429c42a76e055"
Sep 17 21:39:19.379840 ip-10-0-139-36 kubenswrapper[2575]: E0917 21:39:19.379822    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d124a8162b884ae2424e7f435b746adc2f24157ca9869ad373429c42a76e055\": container with ID starting with 9d124a8162b884ae2424e7f435b746adc2f24157ca9869ad373429c42a76e055 not found: ID does not exist" containerID="9d124a8162b884ae2424e7f435b746adc2f24157ca9869ad373429c42a76e055"
Sep 17 21:39:19.379912 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:39:19.379869    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d124a8162b884ae2424e7f435b746adc2f24157ca9869ad373429c42a76e055"} err="failed to get container status \"9d124a8162b884ae2424e7f435b746adc2f24157ca9869ad373429c42a76e055\": rpc error: code = NotFound desc = could not find container \"9d124a8162b884ae2424e7f435b746adc2f24157ca9869ad373429c42a76e055\": container with ID starting with 9d124a8162b884ae2424e7f435b746adc2f24157ca9869ad373429c42a76e055 not found: ID does not exist"
Sep 17 21:39:19.379912 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:39:19.379888    2575 scope.go:117] "RemoveContainer" containerID="69c9b76dd85dc41130cc100923dda34065f94d52fa86b7b997d58f022dd7e2e5"
Sep 17 21:39:19.380113 ip-10-0-139-36 kubenswrapper[2575]: E0917 21:39:19.380096    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69c9b76dd85dc41130cc100923dda34065f94d52fa86b7b997d58f022dd7e2e5\": container with ID starting with 69c9b76dd85dc41130cc100923dda34065f94d52fa86b7b997d58f022dd7e2e5 not found: ID does not exist" containerID="69c9b76dd85dc41130cc100923dda34065f94d52fa86b7b997d58f022dd7e2e5"
Sep 17 21:39:19.380154 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:39:19.380117    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69c9b76dd85dc41130cc100923dda34065f94d52fa86b7b997d58f022dd7e2e5"} err="failed to get container status \"69c9b76dd85dc41130cc100923dda34065f94d52fa86b7b997d58f022dd7e2e5\": rpc error: code = NotFound desc = could not find container \"69c9b76dd85dc41130cc100923dda34065f94d52fa86b7b997d58f022dd7e2e5\": container with ID starting with 69c9b76dd85dc41130cc100923dda34065f94d52fa86b7b997d58f022dd7e2e5 not found: ID does not exist"
Sep 17 21:39:19.385330 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:39:19.385308    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-8mj66"]
Sep 17 21:39:19.389806 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:39:19.389786    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-555fc44b76-8mj66"]
Sep 17 21:39:21.357801 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:39:21.357764    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d81b305-7eb2-4ce4-96e0-08476ca9b9e8" path="/var/lib/kubelet/pods/7d81b305-7eb2-4ce4-96e0-08476ca9b9e8/volumes"
Sep 17 21:39:33.395917 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:39:33.395872    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:39:33.489737 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:39:33.489700    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:39:51.664687 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:39:51.664599    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:39:51.765670 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:39:51.765630    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:39:51.864583 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:39:51.864554    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:39:58.556146 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:39:58.556110    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:40:00.941069 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:40:00.941030    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:40:04.636146 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:40:04.636109    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:40:04.734016 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:40:04.733976    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:40:06.458524 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:40:06.458485    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:40:23.479545 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:40:23.479513    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:40:23.578071 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:40:23.578030    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:40:23.773495 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:40:23.773403    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:40:23.863061 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:40:23.863022    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:40:26.654425 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:40:26.654383    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:40:30.411013 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:40:30.410975    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:40:30.512047 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:40:30.512011    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:40:37.313227 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:40:37.313184    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:40:37.391771 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:40:37.391729    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:40:43.781725 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:40:43.781690    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z"]
Sep 17 21:40:43.782332 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:40:43.782298    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d81b305-7eb2-4ce4-96e0-08476ca9b9e8" containerName="storage-initializer"
Sep 17 21:40:43.782332 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:40:43.782330    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d81b305-7eb2-4ce4-96e0-08476ca9b9e8" containerName="storage-initializer"
Sep 17 21:40:43.782537 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:40:43.782378    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d81b305-7eb2-4ce4-96e0-08476ca9b9e8" containerName="main"
Sep 17 21:40:43.782537 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:40:43.782388    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d81b305-7eb2-4ce4-96e0-08476ca9b9e8" containerName="main"
Sep 17 21:40:43.782537 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:40:43.782480    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d81b305-7eb2-4ce4-96e0-08476ca9b9e8" containerName="main"
Sep 17 21:40:43.785688 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:40:43.785670    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z"
Sep 17 21:40:43.788811 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:40:43.788786    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 17 21:40:43.788962 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:40:43.788813    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 21:40:43.788962 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:40:43.788841    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-k9qhp\""
Sep 17 21:40:43.788962 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:40:43.788840    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 17 21:40:43.788962 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:40:43.788909    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 17 21:40:43.788962 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:40:43.788786    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Sep 17 21:40:43.802143 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:40:43.802120    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z"]
Sep 17 21:40:43.812657 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:40:43.812629    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:40:43.832969 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:40:43.832943    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqrst\" (UniqueName: \"kubernetes.io/projected/fe86af26-9c60-43e5-932a-eb997e4728e2-kube-api-access-vqrst\") pod \"router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z\" (UID: \"fe86af26-9c60-43e5-932a-eb997e4728e2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z"
Sep 17 21:40:43.833088 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:40:43.832980    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe86af26-9c60-43e5-932a-eb997e4728e2-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z\" (UID: \"fe86af26-9c60-43e5-932a-eb997e4728e2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z"
Sep 17 21:40:43.833088 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:40:43.833006    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe86af26-9c60-43e5-932a-eb997e4728e2-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z\" (UID: \"fe86af26-9c60-43e5-932a-eb997e4728e2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z"
Sep 17 21:40:43.833216 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:40:43.833123    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe86af26-9c60-43e5-932a-eb997e4728e2-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z\" (UID: \"fe86af26-9c60-43e5-932a-eb997e4728e2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z"
Sep 17 21:40:43.833216 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:40:43.833175    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe86af26-9c60-43e5-932a-eb997e4728e2-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z\" (UID: \"fe86af26-9c60-43e5-932a-eb997e4728e2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z"
Sep 17 21:40:43.833216 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:40:43.833204    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe86af26-9c60-43e5-932a-eb997e4728e2-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z\" (UID: \"fe86af26-9c60-43e5-932a-eb997e4728e2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z"
Sep 17 21:40:43.833372 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:40:43.833235    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe86af26-9c60-43e5-932a-eb997e4728e2-home\") pod \"router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z\" (UID: \"fe86af26-9c60-43e5-932a-eb997e4728e2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z"
Sep 17 21:40:43.833372 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:40:43.833292    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe86af26-9c60-43e5-932a-eb997e4728e2-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z\" (UID: \"fe86af26-9c60-43e5-932a-eb997e4728e2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z"
Sep 17 21:40:43.934445 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:40:43.934416    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe86af26-9c60-43e5-932a-eb997e4728e2-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z\" (UID: \"fe86af26-9c60-43e5-932a-eb997e4728e2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z"
Sep 17 21:40:43.934603 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:40:43.934453    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe86af26-9c60-43e5-932a-eb997e4728e2-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z\" (UID: \"fe86af26-9c60-43e5-932a-eb997e4728e2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z"
Sep 17 21:40:43.934603 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:40:43.934474    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe86af26-9c60-43e5-932a-eb997e4728e2-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z\" (UID: \"fe86af26-9c60-43e5-932a-eb997e4728e2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z"
Sep 17 21:40:43.934603 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:40:43.934582    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe86af26-9c60-43e5-932a-eb997e4728e2-home\") pod \"router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z\" (UID: \"fe86af26-9c60-43e5-932a-eb997e4728e2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z"
Sep 17 21:40:43.934766 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:40:43.934640    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe86af26-9c60-43e5-932a-eb997e4728e2-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z\" (UID: \"fe86af26-9c60-43e5-932a-eb997e4728e2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z"
Sep 17 21:40:43.934766 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:40:43.934696    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqrst\" (UniqueName: \"kubernetes.io/projected/fe86af26-9c60-43e5-932a-eb997e4728e2-kube-api-access-vqrst\") pod \"router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z\" (UID: \"fe86af26-9c60-43e5-932a-eb997e4728e2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z"
Sep 17 21:40:43.934766 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:40:43.934733    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe86af26-9c60-43e5-932a-eb997e4728e2-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z\" (UID: \"fe86af26-9c60-43e5-932a-eb997e4728e2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z"
Sep 17 21:40:43.934766 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:40:43.934761    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe86af26-9c60-43e5-932a-eb997e4728e2-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z\" (UID: \"fe86af26-9c60-43e5-932a-eb997e4728e2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z"
Sep 17 21:40:43.935018 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:40:43.934904    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe86af26-9c60-43e5-932a-eb997e4728e2-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z\" (UID: \"fe86af26-9c60-43e5-932a-eb997e4728e2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z"
Sep 17 21:40:43.935072 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:40:43.935055    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe86af26-9c60-43e5-932a-eb997e4728e2-home\") pod \"router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z\" (UID: \"fe86af26-9c60-43e5-932a-eb997e4728e2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z"
Sep 17 21:40:43.935163 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:40:43.935137    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe86af26-9c60-43e5-932a-eb997e4728e2-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z\" (UID: \"fe86af26-9c60-43e5-932a-eb997e4728e2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z"
Sep 17 21:40:43.935244 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:40:43.935221    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe86af26-9c60-43e5-932a-eb997e4728e2-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z\" (UID: \"fe86af26-9c60-43e5-932a-eb997e4728e2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z"
Sep 17 21:40:43.935303 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:40:43.935261    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe86af26-9c60-43e5-932a-eb997e4728e2-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z\" (UID: \"fe86af26-9c60-43e5-932a-eb997e4728e2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z"
Sep 17 21:40:43.936922 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:40:43.936897    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe86af26-9c60-43e5-932a-eb997e4728e2-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z\" (UID: \"fe86af26-9c60-43e5-932a-eb997e4728e2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z"
Sep 17 21:40:43.937165 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:40:43.937149    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe86af26-9c60-43e5-932a-eb997e4728e2-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z\" (UID: \"fe86af26-9c60-43e5-932a-eb997e4728e2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z"
Sep 17 21:40:43.942410 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:40:43.942387    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqrst\" (UniqueName: \"kubernetes.io/projected/fe86af26-9c60-43e5-932a-eb997e4728e2-kube-api-access-vqrst\") pod \"router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z\" (UID: \"fe86af26-9c60-43e5-932a-eb997e4728e2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z"
Sep 17 21:40:44.095768 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:40:44.095690    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z"
Sep 17 21:40:44.379268 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:40:44.379228    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z"]
Sep 17 21:40:44.382176 ip-10-0-139-36 kubenswrapper[2575]: W0917 21:40:44.382147    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe86af26_9c60_43e5_932a_eb997e4728e2.slice/crio-c0da2e8893f91bbbb53648c3575743861fd41cb1d255ec632732428bbd61729f WatchSource:0}: Error finding container c0da2e8893f91bbbb53648c3575743861fd41cb1d255ec632732428bbd61729f: Status 404 returned error can't find the container with id c0da2e8893f91bbbb53648c3575743861fd41cb1d255ec632732428bbd61729f
Sep 17 21:40:44.409835 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:40:44.409807    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:40:44.511315 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:40:44.511277    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:40:44.790539 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:40:44.790505    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z" event={"ID":"fe86af26-9c60-43e5-932a-eb997e4728e2","Type":"ContainerStarted","Data":"021dc4a5ab7001f4a96550a46d1213a6f68171120a12575b41b2bdd1d4d762a0"}
Sep 17 21:40:44.791054 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:40:44.790556    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z" event={"ID":"fe86af26-9c60-43e5-932a-eb997e4728e2","Type":"ContainerStarted","Data":"c0da2e8893f91bbbb53648c3575743861fd41cb1d255ec632732428bbd61729f"}
Sep 17 21:40:46.803969 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:40:46.803878    2575 generic.go:356] "Generic (PLEG): container finished" podID="fe86af26-9c60-43e5-932a-eb997e4728e2" containerID="021dc4a5ab7001f4a96550a46d1213a6f68171120a12575b41b2bdd1d4d762a0" exitCode=0
Sep 17 21:40:46.803969 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:40:46.803943    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z" event={"ID":"fe86af26-9c60-43e5-932a-eb997e4728e2","Type":"ContainerDied","Data":"021dc4a5ab7001f4a96550a46d1213a6f68171120a12575b41b2bdd1d4d762a0"}
Sep 17 21:40:47.810987 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:40:47.810950    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z" event={"ID":"fe86af26-9c60-43e5-932a-eb997e4728e2","Type":"ContainerStarted","Data":"db25c6bf49979f0ed2e6fb185fbc95680eab5849c8cce182a4b05dd47de7e202"}
Sep 17 21:40:47.834035 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:40:47.833985    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z" podStartSLOduration=2.757866966 podStartE2EDuration="4.833968818s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.076101852s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:40:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:40:47.83162688 +0000 UTC m=+3112.984071256" watchObservedRunningTime="2026-09-17 21:40:47.833968818 +0000 UTC m=+3112.986413193"
Sep 17 21:40:54.096452 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:40:54.096413    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z"
Sep 17 21:40:54.096976 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:40:54.096514    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z"
Sep 17 21:40:54.098122 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:40:54.098090    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z" podUID="fe86af26-9c60-43e5-932a-eb997e4728e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Sep 17 21:41:04.097014 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:41:04.096964    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z" podUID="fe86af26-9c60-43e5-932a-eb997e4728e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Sep 17 21:41:14.096538 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:41:14.096475    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z" podUID="fe86af26-9c60-43e5-932a-eb997e4728e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Sep 17 21:41:24.096588 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:41:24.096476    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z" podUID="fe86af26-9c60-43e5-932a-eb997e4728e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Sep 17 21:41:34.096243 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:41:34.096183    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z" podUID="fe86af26-9c60-43e5-932a-eb997e4728e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Sep 17 21:41:41.481958 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:41:41.481916    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:41:41.536379 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:41:41.536342    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:41:44.096472 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:41:44.096422    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z" podUID="fe86af26-9c60-43e5-932a-eb997e4728e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Sep 17 21:41:48.418574 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:41:48.418537    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:41:54.096797 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:41:54.096748    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z" podUID="fe86af26-9c60-43e5-932a-eb997e4728e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Sep 17 21:42:04.096096 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:42:04.096045    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z" podUID="fe86af26-9c60-43e5-932a-eb997e4728e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Sep 17 21:42:08.457542 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:42:08.457503    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:42:09.797987 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:42:09.797951    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:42:14.096897 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:42:14.096809    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z" podUID="fe86af26-9c60-43e5-932a-eb997e4728e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Sep 17 21:42:15.530584 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:42:15.530542    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:42:15.601406 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:42:15.601369    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:42:18.207916 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:42:18.207879    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:42:24.096521 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:42:24.096451    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z" podUID="fe86af26-9c60-43e5-932a-eb997e4728e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Sep 17 21:42:32.816654 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:42:32.816608    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:42:34.096275 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:42:34.096220    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z" podUID="fe86af26-9c60-43e5-932a-eb997e4728e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Sep 17 21:42:44.106558 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:42:44.106528    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z"
Sep 17 21:42:44.114136 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:42:44.114095    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z"
Sep 17 21:42:53.291544 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:42:53.291499    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z"]
Sep 17 21:42:53.295168 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:42:53.295135    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:42:53.395328 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:42:53.395295    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:43:08.314418 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:08.314367    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z" podUID="fe86af26-9c60-43e5-932a-eb997e4728e2" containerName="main" containerID="cri-o://db25c6bf49979f0ed2e6fb185fbc95680eab5849c8cce182a4b05dd47de7e202" gracePeriod=45
Sep 17 21:43:09.060645 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:09.060621    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z"
Sep 17 21:43:09.171764 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:09.171679    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe86af26-9c60-43e5-932a-eb997e4728e2-home\") pod \"fe86af26-9c60-43e5-932a-eb997e4728e2\" (UID: \"fe86af26-9c60-43e5-932a-eb997e4728e2\") "
Sep 17 21:43:09.171764 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:09.171730    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe86af26-9c60-43e5-932a-eb997e4728e2-tmp-dir\") pod \"fe86af26-9c60-43e5-932a-eb997e4728e2\" (UID: \"fe86af26-9c60-43e5-932a-eb997e4728e2\") "
Sep 17 21:43:09.171764 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:09.171766    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe86af26-9c60-43e5-932a-eb997e4728e2-kserve-provision-location\") pod \"fe86af26-9c60-43e5-932a-eb997e4728e2\" (UID: \"fe86af26-9c60-43e5-932a-eb997e4728e2\") "
Sep 17 21:43:09.172098 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:09.171805    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe86af26-9c60-43e5-932a-eb997e4728e2-dshm\") pod \"fe86af26-9c60-43e5-932a-eb997e4728e2\" (UID: \"fe86af26-9c60-43e5-932a-eb997e4728e2\") "
Sep 17 21:43:09.172098 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:09.171825    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe86af26-9c60-43e5-932a-eb997e4728e2-cabundle-cert\") pod \"fe86af26-9c60-43e5-932a-eb997e4728e2\" (UID: \"fe86af26-9c60-43e5-932a-eb997e4728e2\") "
Sep 17 21:43:09.172098 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:09.171872    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe86af26-9c60-43e5-932a-eb997e4728e2-model-cache\") pod \"fe86af26-9c60-43e5-932a-eb997e4728e2\" (UID: \"fe86af26-9c60-43e5-932a-eb997e4728e2\") "
Sep 17 21:43:09.172098 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:09.171906    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vqrst\" (UniqueName: \"kubernetes.io/projected/fe86af26-9c60-43e5-932a-eb997e4728e2-kube-api-access-vqrst\") pod \"fe86af26-9c60-43e5-932a-eb997e4728e2\" (UID: \"fe86af26-9c60-43e5-932a-eb997e4728e2\") "
Sep 17 21:43:09.172098 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:09.171934    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe86af26-9c60-43e5-932a-eb997e4728e2-tls-certs\") pod \"fe86af26-9c60-43e5-932a-eb997e4728e2\" (UID: \"fe86af26-9c60-43e5-932a-eb997e4728e2\") "
Sep 17 21:43:09.172373 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:09.172250    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe86af26-9c60-43e5-932a-eb997e4728e2-model-cache" (OuterVolumeSpecName: "model-cache") pod "fe86af26-9c60-43e5-932a-eb997e4728e2" (UID: "fe86af26-9c60-43e5-932a-eb997e4728e2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:43:09.172373 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:09.172347    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe86af26-9c60-43e5-932a-eb997e4728e2-home" (OuterVolumeSpecName: "home") pod "fe86af26-9c60-43e5-932a-eb997e4728e2" (UID: "fe86af26-9c60-43e5-932a-eb997e4728e2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:43:09.172486 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:09.172391    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe86af26-9c60-43e5-932a-eb997e4728e2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fe86af26-9c60-43e5-932a-eb997e4728e2" (UID: "fe86af26-9c60-43e5-932a-eb997e4728e2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:43:09.174335 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:09.174305    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe86af26-9c60-43e5-932a-eb997e4728e2-dshm" (OuterVolumeSpecName: "dshm") pod "fe86af26-9c60-43e5-932a-eb997e4728e2" (UID: "fe86af26-9c60-43e5-932a-eb997e4728e2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:43:09.174484 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:09.174391    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe86af26-9c60-43e5-932a-eb997e4728e2-kube-api-access-vqrst" (OuterVolumeSpecName: "kube-api-access-vqrst") pod "fe86af26-9c60-43e5-932a-eb997e4728e2" (UID: "fe86af26-9c60-43e5-932a-eb997e4728e2"). InnerVolumeSpecName "kube-api-access-vqrst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:43:09.174541 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:09.174480    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe86af26-9c60-43e5-932a-eb997e4728e2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fe86af26-9c60-43e5-932a-eb997e4728e2" (UID: "fe86af26-9c60-43e5-932a-eb997e4728e2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:43:09.184935 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:09.184907    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe86af26-9c60-43e5-932a-eb997e4728e2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fe86af26-9c60-43e5-932a-eb997e4728e2" (UID: "fe86af26-9c60-43e5-932a-eb997e4728e2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:43:09.197207 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:09.197181    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe86af26-9c60-43e5-932a-eb997e4728e2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fe86af26-9c60-43e5-932a-eb997e4728e2" (UID: "fe86af26-9c60-43e5-932a-eb997e4728e2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:43:09.272965 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:09.272927    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe86af26-9c60-43e5-932a-eb997e4728e2-kserve-provision-location\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:43:09.272965 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:09.272959    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe86af26-9c60-43e5-932a-eb997e4728e2-dshm\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:43:09.272965 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:09.272970    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe86af26-9c60-43e5-932a-eb997e4728e2-cabundle-cert\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:43:09.273191 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:09.272980    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe86af26-9c60-43e5-932a-eb997e4728e2-model-cache\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:43:09.273191 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:09.272988    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqrst\" (UniqueName: \"kubernetes.io/projected/fe86af26-9c60-43e5-932a-eb997e4728e2-kube-api-access-vqrst\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:43:09.273191 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:09.272995    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe86af26-9c60-43e5-932a-eb997e4728e2-tls-certs\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:43:09.273191 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:09.273003    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe86af26-9c60-43e5-932a-eb997e4728e2-home\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:43:09.273191 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:09.273010    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe86af26-9c60-43e5-932a-eb997e4728e2-tmp-dir\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:43:09.563566 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:09.563531    2575 generic.go:356] "Generic (PLEG): container finished" podID="fe86af26-9c60-43e5-932a-eb997e4728e2" containerID="db25c6bf49979f0ed2e6fb185fbc95680eab5849c8cce182a4b05dd47de7e202" exitCode=0
Sep 17 21:43:09.563993 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:09.563580    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z"
Sep 17 21:43:09.563993 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:09.563580    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z" event={"ID":"fe86af26-9c60-43e5-932a-eb997e4728e2","Type":"ContainerDied","Data":"db25c6bf49979f0ed2e6fb185fbc95680eab5849c8cce182a4b05dd47de7e202"}
Sep 17 21:43:09.563993 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:09.563705    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z" event={"ID":"fe86af26-9c60-43e5-932a-eb997e4728e2","Type":"ContainerDied","Data":"c0da2e8893f91bbbb53648c3575743861fd41cb1d255ec632732428bbd61729f"}
Sep 17 21:43:09.563993 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:09.563733    2575 scope.go:117] "RemoveContainer" containerID="db25c6bf49979f0ed2e6fb185fbc95680eab5849c8cce182a4b05dd47de7e202"
Sep 17 21:43:09.573446 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:09.573427    2575 scope.go:117] "RemoveContainer" containerID="021dc4a5ab7001f4a96550a46d1213a6f68171120a12575b41b2bdd1d4d762a0"
Sep 17 21:43:09.581306 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:09.581287    2575 scope.go:117] "RemoveContainer" containerID="db25c6bf49979f0ed2e6fb185fbc95680eab5849c8cce182a4b05dd47de7e202"
Sep 17 21:43:09.581600 ip-10-0-139-36 kubenswrapper[2575]: E0917 21:43:09.581581    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db25c6bf49979f0ed2e6fb185fbc95680eab5849c8cce182a4b05dd47de7e202\": container with ID starting with db25c6bf49979f0ed2e6fb185fbc95680eab5849c8cce182a4b05dd47de7e202 not found: ID does not exist" containerID="db25c6bf49979f0ed2e6fb185fbc95680eab5849c8cce182a4b05dd47de7e202"
Sep 17 21:43:09.581691 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:09.581609    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db25c6bf49979f0ed2e6fb185fbc95680eab5849c8cce182a4b05dd47de7e202"} err="failed to get container status \"db25c6bf49979f0ed2e6fb185fbc95680eab5849c8cce182a4b05dd47de7e202\": rpc error: code = NotFound desc = could not find container \"db25c6bf49979f0ed2e6fb185fbc95680eab5849c8cce182a4b05dd47de7e202\": container with ID starting with db25c6bf49979f0ed2e6fb185fbc95680eab5849c8cce182a4b05dd47de7e202 not found: ID does not exist"
Sep 17 21:43:09.581691 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:09.581629    2575 scope.go:117] "RemoveContainer" containerID="021dc4a5ab7001f4a96550a46d1213a6f68171120a12575b41b2bdd1d4d762a0"
Sep 17 21:43:09.582017 ip-10-0-139-36 kubenswrapper[2575]: E0917 21:43:09.581992    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"021dc4a5ab7001f4a96550a46d1213a6f68171120a12575b41b2bdd1d4d762a0\": container with ID starting with 021dc4a5ab7001f4a96550a46d1213a6f68171120a12575b41b2bdd1d4d762a0 not found: ID does not exist" containerID="021dc4a5ab7001f4a96550a46d1213a6f68171120a12575b41b2bdd1d4d762a0"
Sep 17 21:43:09.582096 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:09.582023    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"021dc4a5ab7001f4a96550a46d1213a6f68171120a12575b41b2bdd1d4d762a0"} err="failed to get container status \"021dc4a5ab7001f4a96550a46d1213a6f68171120a12575b41b2bdd1d4d762a0\": rpc error: code = NotFound desc = could not find container \"021dc4a5ab7001f4a96550a46d1213a6f68171120a12575b41b2bdd1d4d762a0\": container with ID starting with 021dc4a5ab7001f4a96550a46d1213a6f68171120a12575b41b2bdd1d4d762a0 not found: ID does not exist"
Sep 17 21:43:09.585162 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:09.585140    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z"]
Sep 17 21:43:09.592482 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:09.592460    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-598d669f99-95t9z"]
Sep 17 21:43:11.357687 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:11.357653    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe86af26-9c60-43e5-932a-eb997e4728e2" path="/var/lib/kubelet/pods/fe86af26-9c60-43e5-932a-eb997e4728e2/volumes"
Sep 17 21:43:43.273909 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:43.273866    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:43:43.381238 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:43.381205    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:43:48.304674 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:48.304631    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:43:51.038165 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:51.038129    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l"]
Sep 17 21:43:51.038570 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:51.038554    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe86af26-9c60-43e5-932a-eb997e4728e2" containerName="main"
Sep 17 21:43:51.038620 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:51.038572    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe86af26-9c60-43e5-932a-eb997e4728e2" containerName="main"
Sep 17 21:43:51.038620 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:51.038582    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe86af26-9c60-43e5-932a-eb997e4728e2" containerName="storage-initializer"
Sep 17 21:43:51.038620 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:51.038588    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe86af26-9c60-43e5-932a-eb997e4728e2" containerName="storage-initializer"
Sep 17 21:43:51.038729 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:51.038651    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe86af26-9c60-43e5-932a-eb997e4728e2" containerName="main"
Sep 17 21:43:51.041777 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:51.041756    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l"
Sep 17 21:43:51.044300 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:51.044274    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"seaweedfs-s3-creds\""
Sep 17 21:43:51.044418 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:51.044322    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"kube-root-ca.crt\""
Sep 17 21:43:51.044418 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:51.044335    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"openshift-service-ca.crt\""
Sep 17 21:43:51.044418 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:51.044357    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"default-dockercfg-vfqrm\""
Sep 17 21:43:51.044418 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:51.044396    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 21:43:51.044648 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:51.044624    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs\""
Sep 17 21:43:51.055871 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:51.055825    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l"]
Sep 17 21:43:51.062241 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:51.062215    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:43:51.137121 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:51.137088    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb57545b-72f9-4662-a2a9-fb2e995f7b91-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l\" (UID: \"eb57545b-72f9-4662-a2a9-fb2e995f7b91\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l"
Sep 17 21:43:51.137286 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:51.137124    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb57545b-72f9-4662-a2a9-fb2e995f7b91-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l\" (UID: \"eb57545b-72f9-4662-a2a9-fb2e995f7b91\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l"
Sep 17 21:43:51.137286 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:51.137150    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb57545b-72f9-4662-a2a9-fb2e995f7b91-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l\" (UID: \"eb57545b-72f9-4662-a2a9-fb2e995f7b91\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l"
Sep 17 21:43:51.137286 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:51.137228    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb57545b-72f9-4662-a2a9-fb2e995f7b91-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l\" (UID: \"eb57545b-72f9-4662-a2a9-fb2e995f7b91\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l"
Sep 17 21:43:51.137286 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:51.137264    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb57545b-72f9-4662-a2a9-fb2e995f7b91-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l\" (UID: \"eb57545b-72f9-4662-a2a9-fb2e995f7b91\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l"
Sep 17 21:43:51.137451 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:51.137341    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb57545b-72f9-4662-a2a9-fb2e995f7b91-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l\" (UID: \"eb57545b-72f9-4662-a2a9-fb2e995f7b91\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l"
Sep 17 21:43:51.137451 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:51.137364    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ffrw\" (UniqueName: \"kubernetes.io/projected/eb57545b-72f9-4662-a2a9-fb2e995f7b91-kube-api-access-6ffrw\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l\" (UID: \"eb57545b-72f9-4662-a2a9-fb2e995f7b91\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l"
Sep 17 21:43:51.137451 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:51.137394    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb57545b-72f9-4662-a2a9-fb2e995f7b91-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l\" (UID: \"eb57545b-72f9-4662-a2a9-fb2e995f7b91\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l"
Sep 17 21:43:51.238565 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:51.238529    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb57545b-72f9-4662-a2a9-fb2e995f7b91-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l\" (UID: \"eb57545b-72f9-4662-a2a9-fb2e995f7b91\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l"
Sep 17 21:43:51.238750 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:51.238571    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb57545b-72f9-4662-a2a9-fb2e995f7b91-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l\" (UID: \"eb57545b-72f9-4662-a2a9-fb2e995f7b91\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l"
Sep 17 21:43:51.238750 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:51.238698    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb57545b-72f9-4662-a2a9-fb2e995f7b91-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l\" (UID: \"eb57545b-72f9-4662-a2a9-fb2e995f7b91\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l"
Sep 17 21:43:51.238750 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:51.238746    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb57545b-72f9-4662-a2a9-fb2e995f7b91-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l\" (UID: \"eb57545b-72f9-4662-a2a9-fb2e995f7b91\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l"
Sep 17 21:43:51.238975 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:51.238782    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb57545b-72f9-4662-a2a9-fb2e995f7b91-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l\" (UID: \"eb57545b-72f9-4662-a2a9-fb2e995f7b91\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l"
Sep 17 21:43:51.238975 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:51.238871    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb57545b-72f9-4662-a2a9-fb2e995f7b91-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l\" (UID: \"eb57545b-72f9-4662-a2a9-fb2e995f7b91\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l"
Sep 17 21:43:51.238975 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:51.238901    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ffrw\" (UniqueName: \"kubernetes.io/projected/eb57545b-72f9-4662-a2a9-fb2e995f7b91-kube-api-access-6ffrw\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l\" (UID: \"eb57545b-72f9-4662-a2a9-fb2e995f7b91\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l"
Sep 17 21:43:51.238975 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:51.238937    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb57545b-72f9-4662-a2a9-fb2e995f7b91-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l\" (UID: \"eb57545b-72f9-4662-a2a9-fb2e995f7b91\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l"
Sep 17 21:43:51.238975 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:51.238950    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb57545b-72f9-4662-a2a9-fb2e995f7b91-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l\" (UID: \"eb57545b-72f9-4662-a2a9-fb2e995f7b91\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l"
Sep 17 21:43:51.239257 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:51.238993    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb57545b-72f9-4662-a2a9-fb2e995f7b91-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l\" (UID: \"eb57545b-72f9-4662-a2a9-fb2e995f7b91\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l"
Sep 17 21:43:51.239257 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:51.239164    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb57545b-72f9-4662-a2a9-fb2e995f7b91-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l\" (UID: \"eb57545b-72f9-4662-a2a9-fb2e995f7b91\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l"
Sep 17 21:43:51.239257 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:51.239209    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb57545b-72f9-4662-a2a9-fb2e995f7b91-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l\" (UID: \"eb57545b-72f9-4662-a2a9-fb2e995f7b91\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l"
Sep 17 21:43:51.239605 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:51.239586    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb57545b-72f9-4662-a2a9-fb2e995f7b91-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l\" (UID: \"eb57545b-72f9-4662-a2a9-fb2e995f7b91\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l"
Sep 17 21:43:51.241132 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:51.241110    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb57545b-72f9-4662-a2a9-fb2e995f7b91-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l\" (UID: \"eb57545b-72f9-4662-a2a9-fb2e995f7b91\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l"
Sep 17 21:43:51.241234 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:51.241207    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb57545b-72f9-4662-a2a9-fb2e995f7b91-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l\" (UID: \"eb57545b-72f9-4662-a2a9-fb2e995f7b91\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l"
Sep 17 21:43:51.247513 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:51.247490    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ffrw\" (UniqueName: \"kubernetes.io/projected/eb57545b-72f9-4662-a2a9-fb2e995f7b91-kube-api-access-6ffrw\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l\" (UID: \"eb57545b-72f9-4662-a2a9-fb2e995f7b91\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l"
Sep 17 21:43:51.353079 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:51.352961    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l"
Sep 17 21:43:51.637037 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:51.636828    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l"]
Sep 17 21:43:51.639949 ip-10-0-139-36 kubenswrapper[2575]: W0917 21:43:51.639921    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb57545b_72f9_4662_a2a9_fb2e995f7b91.slice/crio-46ac64c54f92cc30ae8d36306fc99943433143a5e13c2fa82855d1e864744939 WatchSource:0}: Error finding container 46ac64c54f92cc30ae8d36306fc99943433143a5e13c2fa82855d1e864744939: Status 404 returned error can't find the container with id 46ac64c54f92cc30ae8d36306fc99943433143a5e13c2fa82855d1e864744939
Sep 17 21:43:51.775646 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:51.775599    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l" event={"ID":"eb57545b-72f9-4662-a2a9-fb2e995f7b91","Type":"ContainerStarted","Data":"75141895f0a4f8ab92f2aedb93c28b4e7c2178e05d23167f98cf2889bb9a8139"}
Sep 17 21:43:51.775824 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:51.775669    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l" event={"ID":"eb57545b-72f9-4662-a2a9-fb2e995f7b91","Type":"ContainerStarted","Data":"46ac64c54f92cc30ae8d36306fc99943433143a5e13c2fa82855d1e864744939"}
Sep 17 21:43:54.792407 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:54.792371    2575 generic.go:356] "Generic (PLEG): container finished" podID="eb57545b-72f9-4662-a2a9-fb2e995f7b91" containerID="75141895f0a4f8ab92f2aedb93c28b4e7c2178e05d23167f98cf2889bb9a8139" exitCode=0
Sep 17 21:43:54.792796 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:54.792446    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l" event={"ID":"eb57545b-72f9-4662-a2a9-fb2e995f7b91","Type":"ContainerDied","Data":"75141895f0a4f8ab92f2aedb93c28b4e7c2178e05d23167f98cf2889bb9a8139"}
Sep 17 21:43:55.799646 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:55.799611    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l" event={"ID":"eb57545b-72f9-4662-a2a9-fb2e995f7b91","Type":"ContainerStarted","Data":"1507ef28be360d2d50da4c53af647d8e7aa5edd11fa6eabd3159d9610d7f4cda"}
Sep 17 21:43:55.826193 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:55.826146    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l" podStartSLOduration=2.599466053 podStartE2EDuration="4.826128036s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.226661983s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:43:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:43:55.823803048 +0000 UTC m=+3300.976247418" watchObservedRunningTime="2026-09-17 21:43:55.826128036 +0000 UTC m=+3300.978572412"
Sep 17 21:43:56.353935 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:43:56.353902    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l"
Sep 17 21:44:01.355313 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:44:01.355262    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l" podUID="eb57545b-72f9-4662-a2a9-fb2e995f7b91" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Sep 17 21:44:01.357605 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:44:01.357394    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l"
Sep 17 21:44:06.712284 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:44:06.712240    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:44:09.661312 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:44:09.661274    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:44:11.353531 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:44:11.353491    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l" podUID="eb57545b-72f9-4662-a2a9-fb2e995f7b91" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Sep 17 21:44:11.579642 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:44:11.579606    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:44:17.906424 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:44:17.906337    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:44:21.353445 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:44:21.353395    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l" podUID="eb57545b-72f9-4662-a2a9-fb2e995f7b91" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Sep 17 21:44:26.096076 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:44:26.096038    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:44:26.726920 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:44:26.726877    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:44:26.829735 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:44:26.829676    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:44:26.928954 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:44:26.928918    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:44:27.027200 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:44:27.027111    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:44:27.127469 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:44:27.127431    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:44:27.228192 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:44:27.228155    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:44:27.772704 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:44:27.772666    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:44:27.873547 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:44:27.873488    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:44:29.457620 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:44:29.457579    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:44:29.521149 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:44:29.521118    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:44:30.134860 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:44:30.134800    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:44:31.353943 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:44:31.353908    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l" podUID="eb57545b-72f9-4662-a2a9-fb2e995f7b91" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Sep 17 21:44:33.331517 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:44:33.331484    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:44:33.407320 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:44:33.407287    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:44:33.826625 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:44:33.826591    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:44:33.907202 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:44:33.907168    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:44:41.353827 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:44:41.353788    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l" podUID="eb57545b-72f9-4662-a2a9-fb2e995f7b91" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Sep 17 21:44:51.353419 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:44:51.353372    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l" podUID="eb57545b-72f9-4662-a2a9-fb2e995f7b91" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Sep 17 21:44:53.038791 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:44:53.038755    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:45:01.353592 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:45:01.353542    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l" podUID="eb57545b-72f9-4662-a2a9-fb2e995f7b91" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Sep 17 21:45:11.353407 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:45:11.353368    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l" podUID="eb57545b-72f9-4662-a2a9-fb2e995f7b91" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Sep 17 21:45:21.354206 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:45:21.354164    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l" podUID="eb57545b-72f9-4662-a2a9-fb2e995f7b91" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Sep 17 21:45:31.353544 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:45:31.353493    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l" podUID="eb57545b-72f9-4662-a2a9-fb2e995f7b91" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Sep 17 21:45:41.353626 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:45:41.353578    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l" podUID="eb57545b-72f9-4662-a2a9-fb2e995f7b91" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Sep 17 21:45:51.363506 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:45:51.363426    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l"
Sep 17 21:45:51.371420 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:45:51.371390    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l"
Sep 17 21:45:56.374606 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:45:56.374572    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l"]
Sep 17 21:45:57.046630 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:45:57.046594    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:46:11.396688 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:11.396645    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l" podUID="eb57545b-72f9-4662-a2a9-fb2e995f7b91" containerName="main" containerID="cri-o://1507ef28be360d2d50da4c53af647d8e7aa5edd11fa6eabd3159d9610d7f4cda" gracePeriod=45
Sep 17 21:46:12.146731 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:12.146706    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l"
Sep 17 21:46:12.252565 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:12.252528    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb57545b-72f9-4662-a2a9-fb2e995f7b91-cabundle-cert\") pod \"eb57545b-72f9-4662-a2a9-fb2e995f7b91\" (UID: \"eb57545b-72f9-4662-a2a9-fb2e995f7b91\") "
Sep 17 21:46:12.252565 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:12.252573    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb57545b-72f9-4662-a2a9-fb2e995f7b91-kserve-provision-location\") pod \"eb57545b-72f9-4662-a2a9-fb2e995f7b91\" (UID: \"eb57545b-72f9-4662-a2a9-fb2e995f7b91\") "
Sep 17 21:46:12.252841 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:12.252624    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb57545b-72f9-4662-a2a9-fb2e995f7b91-tmp-dir\") pod \"eb57545b-72f9-4662-a2a9-fb2e995f7b91\" (UID: \"eb57545b-72f9-4662-a2a9-fb2e995f7b91\") "
Sep 17 21:46:12.252841 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:12.252645    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6ffrw\" (UniqueName: \"kubernetes.io/projected/eb57545b-72f9-4662-a2a9-fb2e995f7b91-kube-api-access-6ffrw\") pod \"eb57545b-72f9-4662-a2a9-fb2e995f7b91\" (UID: \"eb57545b-72f9-4662-a2a9-fb2e995f7b91\") "
Sep 17 21:46:12.252841 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:12.252681    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb57545b-72f9-4662-a2a9-fb2e995f7b91-model-cache\") pod \"eb57545b-72f9-4662-a2a9-fb2e995f7b91\" (UID: \"eb57545b-72f9-4662-a2a9-fb2e995f7b91\") "
Sep 17 21:46:12.252841 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:12.252707    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb57545b-72f9-4662-a2a9-fb2e995f7b91-tls-certs\") pod \"eb57545b-72f9-4662-a2a9-fb2e995f7b91\" (UID: \"eb57545b-72f9-4662-a2a9-fb2e995f7b91\") "
Sep 17 21:46:12.252841 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:12.252733    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb57545b-72f9-4662-a2a9-fb2e995f7b91-dshm\") pod \"eb57545b-72f9-4662-a2a9-fb2e995f7b91\" (UID: \"eb57545b-72f9-4662-a2a9-fb2e995f7b91\") "
Sep 17 21:46:12.252841 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:12.252763    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb57545b-72f9-4662-a2a9-fb2e995f7b91-home\") pod \"eb57545b-72f9-4662-a2a9-fb2e995f7b91\" (UID: \"eb57545b-72f9-4662-a2a9-fb2e995f7b91\") "
Sep 17 21:46:12.253172 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:12.253019    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb57545b-72f9-4662-a2a9-fb2e995f7b91-model-cache" (OuterVolumeSpecName: "model-cache") pod "eb57545b-72f9-4662-a2a9-fb2e995f7b91" (UID: "eb57545b-72f9-4662-a2a9-fb2e995f7b91"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:46:12.253172 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:12.253063    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb57545b-72f9-4662-a2a9-fb2e995f7b91-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "eb57545b-72f9-4662-a2a9-fb2e995f7b91" (UID: "eb57545b-72f9-4662-a2a9-fb2e995f7b91"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:46:12.253347 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:12.253325    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb57545b-72f9-4662-a2a9-fb2e995f7b91-cabundle-cert\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:46:12.253422 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:12.253350    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb57545b-72f9-4662-a2a9-fb2e995f7b91-model-cache\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:46:12.253524 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:12.253507    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb57545b-72f9-4662-a2a9-fb2e995f7b91-home" (OuterVolumeSpecName: "home") pod "eb57545b-72f9-4662-a2a9-fb2e995f7b91" (UID: "eb57545b-72f9-4662-a2a9-fb2e995f7b91"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:46:12.255405 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:12.255373    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb57545b-72f9-4662-a2a9-fb2e995f7b91-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "eb57545b-72f9-4662-a2a9-fb2e995f7b91" (UID: "eb57545b-72f9-4662-a2a9-fb2e995f7b91"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:46:12.255627 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:12.255593    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb57545b-72f9-4662-a2a9-fb2e995f7b91-dshm" (OuterVolumeSpecName: "dshm") pod "eb57545b-72f9-4662-a2a9-fb2e995f7b91" (UID: "eb57545b-72f9-4662-a2a9-fb2e995f7b91"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:46:12.255738 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:12.255648    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb57545b-72f9-4662-a2a9-fb2e995f7b91-kube-api-access-6ffrw" (OuterVolumeSpecName: "kube-api-access-6ffrw") pod "eb57545b-72f9-4662-a2a9-fb2e995f7b91" (UID: "eb57545b-72f9-4662-a2a9-fb2e995f7b91"). InnerVolumeSpecName "kube-api-access-6ffrw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:46:12.266078 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:12.266051    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb57545b-72f9-4662-a2a9-fb2e995f7b91-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "eb57545b-72f9-4662-a2a9-fb2e995f7b91" (UID: "eb57545b-72f9-4662-a2a9-fb2e995f7b91"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:46:12.279128 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:12.279103    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb57545b-72f9-4662-a2a9-fb2e995f7b91-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "eb57545b-72f9-4662-a2a9-fb2e995f7b91" (UID: "eb57545b-72f9-4662-a2a9-fb2e995f7b91"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:46:12.353945 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:12.353906    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb57545b-72f9-4662-a2a9-fb2e995f7b91-kserve-provision-location\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:46:12.353945 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:12.353939    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb57545b-72f9-4662-a2a9-fb2e995f7b91-tmp-dir\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:46:12.353945 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:12.353950    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ffrw\" (UniqueName: \"kubernetes.io/projected/eb57545b-72f9-4662-a2a9-fb2e995f7b91-kube-api-access-6ffrw\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:46:12.354187 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:12.353960    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb57545b-72f9-4662-a2a9-fb2e995f7b91-tls-certs\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:46:12.354187 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:12.353968    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb57545b-72f9-4662-a2a9-fb2e995f7b91-dshm\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:46:12.354187 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:12.353976    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb57545b-72f9-4662-a2a9-fb2e995f7b91-home\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:46:12.511224 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:12.511133    2575 generic.go:356] "Generic (PLEG): container finished" podID="eb57545b-72f9-4662-a2a9-fb2e995f7b91" containerID="1507ef28be360d2d50da4c53af647d8e7aa5edd11fa6eabd3159d9610d7f4cda" exitCode=0
Sep 17 21:46:12.511224 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:12.511211    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l"
Sep 17 21:46:12.511725 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:12.511220    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l" event={"ID":"eb57545b-72f9-4662-a2a9-fb2e995f7b91","Type":"ContainerDied","Data":"1507ef28be360d2d50da4c53af647d8e7aa5edd11fa6eabd3159d9610d7f4cda"}
Sep 17 21:46:12.511725 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:12.511306    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l" event={"ID":"eb57545b-72f9-4662-a2a9-fb2e995f7b91","Type":"ContainerDied","Data":"46ac64c54f92cc30ae8d36306fc99943433143a5e13c2fa82855d1e864744939"}
Sep 17 21:46:12.511725 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:12.511331    2575 scope.go:117] "RemoveContainer" containerID="1507ef28be360d2d50da4c53af647d8e7aa5edd11fa6eabd3159d9610d7f4cda"
Sep 17 21:46:12.521313 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:12.521297    2575 scope.go:117] "RemoveContainer" containerID="75141895f0a4f8ab92f2aedb93c28b4e7c2178e05d23167f98cf2889bb9a8139"
Sep 17 21:46:12.532220 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:12.532200    2575 scope.go:117] "RemoveContainer" containerID="1507ef28be360d2d50da4c53af647d8e7aa5edd11fa6eabd3159d9610d7f4cda"
Sep 17 21:46:12.532533 ip-10-0-139-36 kubenswrapper[2575]: E0917 21:46:12.532516    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1507ef28be360d2d50da4c53af647d8e7aa5edd11fa6eabd3159d9610d7f4cda\": container with ID starting with 1507ef28be360d2d50da4c53af647d8e7aa5edd11fa6eabd3159d9610d7f4cda not found: ID does not exist" containerID="1507ef28be360d2d50da4c53af647d8e7aa5edd11fa6eabd3159d9610d7f4cda"
Sep 17 21:46:12.532587 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:12.532542    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1507ef28be360d2d50da4c53af647d8e7aa5edd11fa6eabd3159d9610d7f4cda"} err="failed to get container status \"1507ef28be360d2d50da4c53af647d8e7aa5edd11fa6eabd3159d9610d7f4cda\": rpc error: code = NotFound desc = could not find container \"1507ef28be360d2d50da4c53af647d8e7aa5edd11fa6eabd3159d9610d7f4cda\": container with ID starting with 1507ef28be360d2d50da4c53af647d8e7aa5edd11fa6eabd3159d9610d7f4cda not found: ID does not exist"
Sep 17 21:46:12.532587 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:12.532565    2575 scope.go:117] "RemoveContainer" containerID="75141895f0a4f8ab92f2aedb93c28b4e7c2178e05d23167f98cf2889bb9a8139"
Sep 17 21:46:12.532810 ip-10-0-139-36 kubenswrapper[2575]: E0917 21:46:12.532792    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75141895f0a4f8ab92f2aedb93c28b4e7c2178e05d23167f98cf2889bb9a8139\": container with ID starting with 75141895f0a4f8ab92f2aedb93c28b4e7c2178e05d23167f98cf2889bb9a8139 not found: ID does not exist" containerID="75141895f0a4f8ab92f2aedb93c28b4e7c2178e05d23167f98cf2889bb9a8139"
Sep 17 21:46:12.532878 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:12.532815    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75141895f0a4f8ab92f2aedb93c28b4e7c2178e05d23167f98cf2889bb9a8139"} err="failed to get container status \"75141895f0a4f8ab92f2aedb93c28b4e7c2178e05d23167f98cf2889bb9a8139\": rpc error: code = NotFound desc = could not find container \"75141895f0a4f8ab92f2aedb93c28b4e7c2178e05d23167f98cf2889bb9a8139\": container with ID starting with 75141895f0a4f8ab92f2aedb93c28b4e7c2178e05d23167f98cf2889bb9a8139 not found: ID does not exist"
Sep 17 21:46:12.539737 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:12.539713    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l"]
Sep 17 21:46:12.549070 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:12.549043    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6ccfff87864pp8l"]
Sep 17 21:46:13.357059 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:13.357022    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb57545b-72f9-4662-a2a9-fb2e995f7b91" path="/var/lib/kubelet/pods/eb57545b-72f9-4662-a2a9-fb2e995f7b91/volumes"
Sep 17 21:46:16.372632 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:16.372584    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:46:16.468820 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:16.468785    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:46:23.236680 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:23.236640    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:46:29.888116 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:29.888080    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:46:30.023188 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:30.023152    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:46:30.246264 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:30.246218    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:46:30.413182 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:30.413142    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:46:30.523429 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:30.523349    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:46:30.597757 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:30.597726    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 17 21:46:30.598135 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:30.598120    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb57545b-72f9-4662-a2a9-fb2e995f7b91" containerName="storage-initializer"
Sep 17 21:46:30.598135 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:30.598136    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb57545b-72f9-4662-a2a9-fb2e995f7b91" containerName="storage-initializer"
Sep 17 21:46:30.598297 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:30.598172    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb57545b-72f9-4662-a2a9-fb2e995f7b91" containerName="main"
Sep 17 21:46:30.598297 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:30.598178    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb57545b-72f9-4662-a2a9-fb2e995f7b91" containerName="main"
Sep 17 21:46:30.598297 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:30.598243    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb57545b-72f9-4662-a2a9-fb2e995f7b91" containerName="main"
Sep 17 21:46:30.603114 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:30.603077    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 21:46:30.606140 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:30.606118    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 21:46:30.606287 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:30.606216    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 17 21:46:30.606548 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:30.606533    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 17 21:46:30.606948 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:30.606932    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-2dr27\""
Sep 17 21:46:30.607010 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:30.606937    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-xcc7f\""
Sep 17 21:46:30.607010 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:30.606982    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 17 21:46:30.613945 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:30.613928    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Sep 17 21:46:30.620692 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:30.620667    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 17 21:46:30.674494 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:30.674460    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:46:30.717912 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:30.717882    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c708d98-1452-4821-820f-0d753a1c1192-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"9c708d98-1452-4821-820f-0d753a1c1192\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 21:46:30.717912 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:30.717913    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c708d98-1452-4821-820f-0d753a1c1192-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"9c708d98-1452-4821-820f-0d753a1c1192\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 21:46:30.718092 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:30.717932    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpxf7\" (UniqueName: \"kubernetes.io/projected/9c708d98-1452-4821-820f-0d753a1c1192-kube-api-access-wpxf7\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"9c708d98-1452-4821-820f-0d753a1c1192\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 21:46:30.718092 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:30.717952    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c708d98-1452-4821-820f-0d753a1c1192-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"9c708d98-1452-4821-820f-0d753a1c1192\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 21:46:30.718092 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:30.718040    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c708d98-1452-4821-820f-0d753a1c1192-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"9c708d98-1452-4821-820f-0d753a1c1192\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 21:46:30.718092 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:30.718065    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c708d98-1452-4821-820f-0d753a1c1192-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"9c708d98-1452-4821-820f-0d753a1c1192\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 21:46:30.718226 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:30.718131    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c708d98-1452-4821-820f-0d753a1c1192-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"9c708d98-1452-4821-820f-0d753a1c1192\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 21:46:30.718226 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:30.718158    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c708d98-1452-4821-820f-0d753a1c1192-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"9c708d98-1452-4821-820f-0d753a1c1192\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 21:46:30.819588 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:30.819503    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c708d98-1452-4821-820f-0d753a1c1192-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"9c708d98-1452-4821-820f-0d753a1c1192\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 21:46:30.819588 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:30.819553    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c708d98-1452-4821-820f-0d753a1c1192-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"9c708d98-1452-4821-820f-0d753a1c1192\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 21:46:30.819807 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:30.819618    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c708d98-1452-4821-820f-0d753a1c1192-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"9c708d98-1452-4821-820f-0d753a1c1192\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 21:46:30.819807 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:30.819642    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c708d98-1452-4821-820f-0d753a1c1192-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"9c708d98-1452-4821-820f-0d753a1c1192\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 21:46:30.819807 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:30.819666    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpxf7\" (UniqueName: \"kubernetes.io/projected/9c708d98-1452-4821-820f-0d753a1c1192-kube-api-access-wpxf7\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"9c708d98-1452-4821-820f-0d753a1c1192\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 21:46:30.819807 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:30.819695    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c708d98-1452-4821-820f-0d753a1c1192-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"9c708d98-1452-4821-820f-0d753a1c1192\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 21:46:30.819807 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:30.819736    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c708d98-1452-4821-820f-0d753a1c1192-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"9c708d98-1452-4821-820f-0d753a1c1192\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 21:46:30.819807 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:30.819759    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c708d98-1452-4821-820f-0d753a1c1192-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"9c708d98-1452-4821-820f-0d753a1c1192\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 21:46:30.820226 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:30.819905    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c708d98-1452-4821-820f-0d753a1c1192-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"9c708d98-1452-4821-820f-0d753a1c1192\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 21:46:30.820226 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:30.819942    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c708d98-1452-4821-820f-0d753a1c1192-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"9c708d98-1452-4821-820f-0d753a1c1192\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 21:46:30.820226 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:30.820130    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c708d98-1452-4821-820f-0d753a1c1192-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"9c708d98-1452-4821-820f-0d753a1c1192\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 21:46:30.820350 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:30.820250    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c708d98-1452-4821-820f-0d753a1c1192-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"9c708d98-1452-4821-820f-0d753a1c1192\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 21:46:30.820350 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:30.820298    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c708d98-1452-4821-820f-0d753a1c1192-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"9c708d98-1452-4821-820f-0d753a1c1192\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 21:46:30.822073 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:30.822050    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c708d98-1452-4821-820f-0d753a1c1192-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"9c708d98-1452-4821-820f-0d753a1c1192\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 21:46:30.822192 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:30.822165    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c708d98-1452-4821-820f-0d753a1c1192-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"9c708d98-1452-4821-820f-0d753a1c1192\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 21:46:30.827477 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:30.827446    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpxf7\" (UniqueName: \"kubernetes.io/projected/9c708d98-1452-4821-820f-0d753a1c1192-kube-api-access-wpxf7\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"9c708d98-1452-4821-820f-0d753a1c1192\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 21:46:30.912857 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:30.912813    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 21:46:31.197075 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:31.197044    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 17 21:46:31.199871 ip-10-0-139-36 kubenswrapper[2575]: W0917 21:46:31.199831    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c708d98_1452_4821_820f_0d753a1c1192.slice/crio-3f3a0caf47881f2ac143b3de4bb838bbfcb5874277e6b05cc6277df916c14b65 WatchSource:0}: Error finding container 3f3a0caf47881f2ac143b3de4bb838bbfcb5874277e6b05cc6277df916c14b65: Status 404 returned error can't find the container with id 3f3a0caf47881f2ac143b3de4bb838bbfcb5874277e6b05cc6277df916c14b65
Sep 17 21:46:31.618669 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:31.618568    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"9c708d98-1452-4821-820f-0d753a1c1192","Type":"ContainerStarted","Data":"02e8d70fb35e194ff48705abe83a7602ae6ea7e9dcad32abda96a44410326c48"}
Sep 17 21:46:31.619028 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:31.619004    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"9c708d98-1452-4821-820f-0d753a1c1192","Type":"ContainerStarted","Data":"3f3a0caf47881f2ac143b3de4bb838bbfcb5874277e6b05cc6277df916c14b65"}
Sep 17 21:46:33.632669 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:33.632636    2575 generic.go:356] "Generic (PLEG): container finished" podID="9c708d98-1452-4821-820f-0d753a1c1192" containerID="02e8d70fb35e194ff48705abe83a7602ae6ea7e9dcad32abda96a44410326c48" exitCode=0
Sep 17 21:46:33.633181 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:33.632698    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"9c708d98-1452-4821-820f-0d753a1c1192","Type":"ContainerDied","Data":"02e8d70fb35e194ff48705abe83a7602ae6ea7e9dcad32abda96a44410326c48"}
Sep 17 21:46:34.643275 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:34.643239    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"9c708d98-1452-4821-820f-0d753a1c1192","Type":"ContainerStarted","Data":"b6bc26ff28e1c3855ad82bb5b7cb8e065d09527eb8f7867e6256e106266ce9be"}
Sep 17 21:46:34.673941 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:34.673874    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podStartSLOduration=2.504198327 podStartE2EDuration="4.673840306s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.169641979s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:46:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:46:34.672901989 +0000 UTC m=+3459.825346365" watchObservedRunningTime="2026-09-17 21:46:34.673840306 +0000 UTC m=+3459.826284709"
Sep 17 21:46:34.914014 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:34.913915    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 21:46:40.912979 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:40.912931    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 21:46:40.914771 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:40.914736    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="9c708d98-1452-4821-820f-0d753a1c1192" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Sep 17 21:46:43.545750 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:43.545706    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:46:50.913487 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:46:50.913438    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="9c708d98-1452-4821-820f-0d753a1c1192" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Sep 17 21:47:00.914161 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:47:00.914110    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="9c708d98-1452-4821-820f-0d753a1c1192" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Sep 17 21:47:10.914090 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:47:10.914037    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="9c708d98-1452-4821-820f-0d753a1c1192" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Sep 17 21:47:10.979915 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:47:10.979877    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:47:20.913762 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:47:20.913662    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="9c708d98-1452-4821-820f-0d753a1c1192" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Sep 17 21:47:30.913510 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:47:30.913460    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="9c708d98-1452-4821-820f-0d753a1c1192" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Sep 17 21:47:31.039116 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:47:31.039073    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:47:31.142207 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:47:31.142170    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:47:31.243432 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:47:31.243395    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:47:31.321318 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:47:31.321281    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:47:31.410371 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:47:31.410340    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:47:33.559447 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:47:33.559402    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:47:33.662380 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:47:33.662350    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:47:34.426498 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:47:34.426432    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:47:36.999307 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:47:36.999276    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:47:37.098552 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:47:37.098524    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:47:37.202953 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:47:37.202919    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:47:37.301051 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:47:37.300971    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:47:37.706870 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:47:37.706815    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:47:40.914085 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:47:40.914042    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="9c708d98-1452-4821-820f-0d753a1c1192" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Sep 17 21:47:42.048784 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:47:42.048747    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:47:43.897622 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:47:43.897590    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:47:43.999642 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:47:43.999605    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:47:50.913376 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:47:50.913332    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="9c708d98-1452-4821-820f-0d753a1c1192" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Sep 17 21:47:59.834741 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:47:59.834705    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:47:59.907697 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:47:59.907648    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:48:00.913930 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:48:00.913885    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="9c708d98-1452-4821-820f-0d753a1c1192" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Sep 17 21:48:03.022924 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:48:03.022883    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:48:10.923581 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:48:10.923544    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 21:48:10.931232 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:48:10.931204    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 21:48:30.954088 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:48:30.954050    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:48:50.657313 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:48:50.657236    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:48:50.679576 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:48:50.679543    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 17 21:48:50.754551 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:48:50.754513    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:49:03.343484 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:03.343446    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:49:03.421075 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:03.421042    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:49:05.701609 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:05.701538    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="9c708d98-1452-4821-820f-0d753a1c1192" containerName="main" containerID="cri-o://b6bc26ff28e1c3855ad82bb5b7cb8e065d09527eb8f7867e6256e106266ce9be" gracePeriod=45
Sep 17 21:49:05.924673 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:05.924626    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="9c708d98-1452-4821-820f-0d753a1c1192" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Sep 17 21:49:06.421320 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:06.421287    2575 generic.go:356] "Generic (PLEG): container finished" podID="9c708d98-1452-4821-820f-0d753a1c1192" containerID="b6bc26ff28e1c3855ad82bb5b7cb8e065d09527eb8f7867e6256e106266ce9be" exitCode=0
Sep 17 21:49:06.421481 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:06.421360    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"9c708d98-1452-4821-820f-0d753a1c1192","Type":"ContainerDied","Data":"b6bc26ff28e1c3855ad82bb5b7cb8e065d09527eb8f7867e6256e106266ce9be"}
Sep 17 21:49:06.462206 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:06.462179    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 21:49:06.618981 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:06.618898    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c708d98-1452-4821-820f-0d753a1c1192-model-cache\") pod \"9c708d98-1452-4821-820f-0d753a1c1192\" (UID: \"9c708d98-1452-4821-820f-0d753a1c1192\") "
Sep 17 21:49:06.618981 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:06.618943    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c708d98-1452-4821-820f-0d753a1c1192-kserve-provision-location\") pod \"9c708d98-1452-4821-820f-0d753a1c1192\" (UID: \"9c708d98-1452-4821-820f-0d753a1c1192\") "
Sep 17 21:49:06.618981 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:06.618965    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c708d98-1452-4821-820f-0d753a1c1192-tmp-dir\") pod \"9c708d98-1452-4821-820f-0d753a1c1192\" (UID: \"9c708d98-1452-4821-820f-0d753a1c1192\") "
Sep 17 21:49:06.619263 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:06.618999    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c708d98-1452-4821-820f-0d753a1c1192-home\") pod \"9c708d98-1452-4821-820f-0d753a1c1192\" (UID: \"9c708d98-1452-4821-820f-0d753a1c1192\") "
Sep 17 21:49:06.619263 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:06.619043    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wpxf7\" (UniqueName: \"kubernetes.io/projected/9c708d98-1452-4821-820f-0d753a1c1192-kube-api-access-wpxf7\") pod \"9c708d98-1452-4821-820f-0d753a1c1192\" (UID: \"9c708d98-1452-4821-820f-0d753a1c1192\") "
Sep 17 21:49:06.619263 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:06.619070    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c708d98-1452-4821-820f-0d753a1c1192-cabundle-cert\") pod \"9c708d98-1452-4821-820f-0d753a1c1192\" (UID: \"9c708d98-1452-4821-820f-0d753a1c1192\") "
Sep 17 21:49:06.619263 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:06.619106    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c708d98-1452-4821-820f-0d753a1c1192-dshm\") pod \"9c708d98-1452-4821-820f-0d753a1c1192\" (UID: \"9c708d98-1452-4821-820f-0d753a1c1192\") "
Sep 17 21:49:06.619263 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:06.619157    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c708d98-1452-4821-820f-0d753a1c1192-tls-certs\") pod \"9c708d98-1452-4821-820f-0d753a1c1192\" (UID: \"9c708d98-1452-4821-820f-0d753a1c1192\") "
Sep 17 21:49:06.619534 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:06.619353    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c708d98-1452-4821-820f-0d753a1c1192-model-cache" (OuterVolumeSpecName: "model-cache") pod "9c708d98-1452-4821-820f-0d753a1c1192" (UID: "9c708d98-1452-4821-820f-0d753a1c1192"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:49:06.619712 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:06.619681    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c708d98-1452-4821-820f-0d753a1c1192-home" (OuterVolumeSpecName: "home") pod "9c708d98-1452-4821-820f-0d753a1c1192" (UID: "9c708d98-1452-4821-820f-0d753a1c1192"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:49:06.619797 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:06.619770    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c708d98-1452-4821-820f-0d753a1c1192-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9c708d98-1452-4821-820f-0d753a1c1192" (UID: "9c708d98-1452-4821-820f-0d753a1c1192"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:49:06.622064 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:06.622030    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c708d98-1452-4821-820f-0d753a1c1192-dshm" (OuterVolumeSpecName: "dshm") pod "9c708d98-1452-4821-820f-0d753a1c1192" (UID: "9c708d98-1452-4821-820f-0d753a1c1192"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:49:06.622181 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:06.622088    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c708d98-1452-4821-820f-0d753a1c1192-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9c708d98-1452-4821-820f-0d753a1c1192" (UID: "9c708d98-1452-4821-820f-0d753a1c1192"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:49:06.622181 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:06.622156    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c708d98-1452-4821-820f-0d753a1c1192-kube-api-access-wpxf7" (OuterVolumeSpecName: "kube-api-access-wpxf7") pod "9c708d98-1452-4821-820f-0d753a1c1192" (UID: "9c708d98-1452-4821-820f-0d753a1c1192"). InnerVolumeSpecName "kube-api-access-wpxf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:49:06.632807 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:06.632783    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c708d98-1452-4821-820f-0d753a1c1192-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9c708d98-1452-4821-820f-0d753a1c1192" (UID: "9c708d98-1452-4821-820f-0d753a1c1192"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:49:06.644273 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:06.644243    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c708d98-1452-4821-820f-0d753a1c1192-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9c708d98-1452-4821-820f-0d753a1c1192" (UID: "9c708d98-1452-4821-820f-0d753a1c1192"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:49:06.720545 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:06.720506    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c708d98-1452-4821-820f-0d753a1c1192-model-cache\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:49:06.720545 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:06.720540    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c708d98-1452-4821-820f-0d753a1c1192-kserve-provision-location\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:49:06.721003 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:06.720554    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c708d98-1452-4821-820f-0d753a1c1192-tmp-dir\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:49:06.721003 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:06.720570    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c708d98-1452-4821-820f-0d753a1c1192-home\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:49:06.721003 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:06.720581    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpxf7\" (UniqueName: \"kubernetes.io/projected/9c708d98-1452-4821-820f-0d753a1c1192-kube-api-access-wpxf7\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:49:06.721003 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:06.720590    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c708d98-1452-4821-820f-0d753a1c1192-cabundle-cert\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:49:06.721003 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:06.720598    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c708d98-1452-4821-820f-0d753a1c1192-dshm\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:49:06.721003 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:06.720605    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c708d98-1452-4821-820f-0d753a1c1192-tls-certs\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:49:07.428695 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:07.428665    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 21:49:07.428695 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:07.428690    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"9c708d98-1452-4821-820f-0d753a1c1192","Type":"ContainerDied","Data":"3f3a0caf47881f2ac143b3de4bb838bbfcb5874277e6b05cc6277df916c14b65"}
Sep 17 21:49:07.428970 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:07.428754    2575 scope.go:117] "RemoveContainer" containerID="b6bc26ff28e1c3855ad82bb5b7cb8e065d09527eb8f7867e6256e106266ce9be"
Sep 17 21:49:07.437445 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:07.437423    2575 scope.go:117] "RemoveContainer" containerID="02e8d70fb35e194ff48705abe83a7602ae6ea7e9dcad32abda96a44410326c48"
Sep 17 21:49:07.450591 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:07.450567    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 17 21:49:07.456140 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:07.456117    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 17 21:49:09.357038 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:09.357005    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c708d98-1452-4821-820f-0d753a1c1192" path="/var/lib/kubelet/pods/9c708d98-1452-4821-820f-0d753a1c1192/volumes"
Sep 17 21:49:40.714569 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:40.714530    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:49:40.816555 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:40.816518    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:49:46.210288 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:46.210250    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-h224t"]
Sep 17 21:49:46.210711 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:46.210605    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c708d98-1452-4821-820f-0d753a1c1192" containerName="main"
Sep 17 21:49:46.210711 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:46.210617    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c708d98-1452-4821-820f-0d753a1c1192" containerName="main"
Sep 17 21:49:46.210711 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:46.210640    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c708d98-1452-4821-820f-0d753a1c1192" containerName="storage-initializer"
Sep 17 21:49:46.210711 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:46.210646    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c708d98-1452-4821-820f-0d753a1c1192" containerName="storage-initializer"
Sep 17 21:49:46.210711 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:46.210700    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c708d98-1452-4821-820f-0d753a1c1192" containerName="main"
Sep 17 21:49:46.213652 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:46.213631    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-h224t"
Sep 17 21:49:46.216722 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:46.216702    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Sep 17 21:49:46.216832 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:46.216768    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Sep 17 21:49:46.216832 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:46.216773    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-r2nn2\""
Sep 17 21:49:46.216943 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:46.216810    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Sep 17 21:49:46.225181 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:46.225158    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-h224t"]
Sep 17 21:49:46.306822 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:46.306791    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:49:46.363546 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:46.363519    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2d2v\" (UniqueName: \"kubernetes.io/projected/f66bba6c-f7e4-4fbf-9150-a39d5fd382e6-kube-api-access-x2d2v\") pod \"scheduler-inline-config-test-kserve-8655bc757-h224t\" (UID: \"f66bba6c-f7e4-4fbf-9150-a39d5fd382e6\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-h224t"
Sep 17 21:49:46.363728 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:46.363573    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f66bba6c-f7e4-4fbf-9150-a39d5fd382e6-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-h224t\" (UID: \"f66bba6c-f7e4-4fbf-9150-a39d5fd382e6\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-h224t"
Sep 17 21:49:46.363728 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:46.363622    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f66bba6c-f7e4-4fbf-9150-a39d5fd382e6-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-h224t\" (UID: \"f66bba6c-f7e4-4fbf-9150-a39d5fd382e6\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-h224t"
Sep 17 21:49:46.363728 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:46.363636    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f66bba6c-f7e4-4fbf-9150-a39d5fd382e6-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-h224t\" (UID: \"f66bba6c-f7e4-4fbf-9150-a39d5fd382e6\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-h224t"
Sep 17 21:49:46.363728 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:46.363716    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f66bba6c-f7e4-4fbf-9150-a39d5fd382e6-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-h224t\" (UID: \"f66bba6c-f7e4-4fbf-9150-a39d5fd382e6\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-h224t"
Sep 17 21:49:46.363974 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:46.363755    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f66bba6c-f7e4-4fbf-9150-a39d5fd382e6-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-h224t\" (UID: \"f66bba6c-f7e4-4fbf-9150-a39d5fd382e6\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-h224t"
Sep 17 21:49:46.464479 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:46.464383    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f66bba6c-f7e4-4fbf-9150-a39d5fd382e6-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-h224t\" (UID: \"f66bba6c-f7e4-4fbf-9150-a39d5fd382e6\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-h224t"
Sep 17 21:49:46.464479 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:46.464442    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f66bba6c-f7e4-4fbf-9150-a39d5fd382e6-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-h224t\" (UID: \"f66bba6c-f7e4-4fbf-9150-a39d5fd382e6\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-h224t"
Sep 17 21:49:46.464479 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:46.464459    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f66bba6c-f7e4-4fbf-9150-a39d5fd382e6-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-h224t\" (UID: \"f66bba6c-f7e4-4fbf-9150-a39d5fd382e6\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-h224t"
Sep 17 21:49:46.464479 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:46.464483    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f66bba6c-f7e4-4fbf-9150-a39d5fd382e6-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-h224t\" (UID: \"f66bba6c-f7e4-4fbf-9150-a39d5fd382e6\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-h224t"
Sep 17 21:49:46.464827 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:46.464500    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f66bba6c-f7e4-4fbf-9150-a39d5fd382e6-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-h224t\" (UID: \"f66bba6c-f7e4-4fbf-9150-a39d5fd382e6\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-h224t"
Sep 17 21:49:46.464827 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:46.464535    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2d2v\" (UniqueName: \"kubernetes.io/projected/f66bba6c-f7e4-4fbf-9150-a39d5fd382e6-kube-api-access-x2d2v\") pod \"scheduler-inline-config-test-kserve-8655bc757-h224t\" (UID: \"f66bba6c-f7e4-4fbf-9150-a39d5fd382e6\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-h224t"
Sep 17 21:49:46.465044 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:46.464928    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f66bba6c-f7e4-4fbf-9150-a39d5fd382e6-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-h224t\" (UID: \"f66bba6c-f7e4-4fbf-9150-a39d5fd382e6\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-h224t"
Sep 17 21:49:46.465044 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:46.464969    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f66bba6c-f7e4-4fbf-9150-a39d5fd382e6-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-h224t\" (UID: \"f66bba6c-f7e4-4fbf-9150-a39d5fd382e6\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-h224t"
Sep 17 21:49:46.465185 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:46.465170    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f66bba6c-f7e4-4fbf-9150-a39d5fd382e6-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-h224t\" (UID: \"f66bba6c-f7e4-4fbf-9150-a39d5fd382e6\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-h224t"
Sep 17 21:49:46.466830 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:46.466809    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f66bba6c-f7e4-4fbf-9150-a39d5fd382e6-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-h224t\" (UID: \"f66bba6c-f7e4-4fbf-9150-a39d5fd382e6\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-h224t"
Sep 17 21:49:46.467133 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:46.467115    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f66bba6c-f7e4-4fbf-9150-a39d5fd382e6-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-h224t\" (UID: \"f66bba6c-f7e4-4fbf-9150-a39d5fd382e6\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-h224t"
Sep 17 21:49:46.473138 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:46.473115    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2d2v\" (UniqueName: \"kubernetes.io/projected/f66bba6c-f7e4-4fbf-9150-a39d5fd382e6-kube-api-access-x2d2v\") pod \"scheduler-inline-config-test-kserve-8655bc757-h224t\" (UID: \"f66bba6c-f7e4-4fbf-9150-a39d5fd382e6\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-h224t"
Sep 17 21:49:46.523979 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:46.523951    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-h224t"
Sep 17 21:49:46.571399 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:46.571364    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:49:46.816490 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:46.816407    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-h224t"]
Sep 17 21:49:46.819821 ip-10-0-139-36 kubenswrapper[2575]: W0917 21:49:46.819788    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf66bba6c_f7e4_4fbf_9150_a39d5fd382e6.slice/crio-dbcb9f3afefdd2e202c97b4771342dc1a4af496ce0cef9d483705430a06bf44f WatchSource:0}: Error finding container dbcb9f3afefdd2e202c97b4771342dc1a4af496ce0cef9d483705430a06bf44f: Status 404 returned error can't find the container with id dbcb9f3afefdd2e202c97b4771342dc1a4af496ce0cef9d483705430a06bf44f
Sep 17 21:49:47.626217 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:47.626176    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-h224t" event={"ID":"f66bba6c-f7e4-4fbf-9150-a39d5fd382e6","Type":"ContainerStarted","Data":"a2996b3e227481dfe81862c7663903a8d8debd3f10cd3ac5fe875d32dff68a02"}
Sep 17 21:49:47.626686 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:47.626247    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-h224t" event={"ID":"f66bba6c-f7e4-4fbf-9150-a39d5fd382e6","Type":"ContainerStarted","Data":"dbcb9f3afefdd2e202c97b4771342dc1a4af496ce0cef9d483705430a06bf44f"}
Sep 17 21:49:47.686336 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:47.686283    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-h224t" podStartSLOduration=1.686264343 podStartE2EDuration="1.686264343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:49:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:49:47.683564655 +0000 UTC m=+3652.836009033" watchObservedRunningTime="2026-09-17 21:49:47.686264343 +0000 UTC m=+3652.838708719"
Sep 17 21:49:48.524312 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:48.524276    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-h224t"
Sep 17 21:49:48.859420 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:48.859329    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:49:55.671746 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:55.671706    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:49:55.768084 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:55.768051    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:49:56.524697 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:56.524664    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-h224t"
Sep 17 21:49:56.536996 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:56.536969    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-h224t"
Sep 17 21:49:56.681759 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:49:56.681730    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-h224t"
Sep 17 21:50:03.073690 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:03.073652    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:50:04.190744 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:04.190711    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:50:19.799010 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:19.798932    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:50:19.837320 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:19.837283    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-h224t"]
Sep 17 21:50:19.846977 ip-10-0-139-36 kubenswrapper[2575]: E0917 21:50:19.846942    2575 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 21:50:19.846977 ip-10-0-139-36 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:50:19.846977 ip-10-0-139-36 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 17 21:50:19.846977 ip-10-0-139-36 kubenswrapper[2575]:  > containerID="a2996b3e227481dfe81862c7663903a8d8debd3f10cd3ac5fe875d32dff68a02" cmd=["/bin/sleep","15"]
Sep 17 21:50:19.847278 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:19.846994    2575 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-h224t" message="" err=<
Sep 17 21:50:19.847278 ip-10-0-139-36 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:50:19.847278 ip-10-0-139-36 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 17 21:50:19.847278 ip-10-0-139-36 kubenswrapper[2575]:  >
Sep 17 21:50:19.847278 ip-10-0-139-36 kubenswrapper[2575]: E0917 21:50:19.847008    2575 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 21:50:19.847278 ip-10-0-139-36 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:50:19.847278 ip-10-0-139-36 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 17 21:50:19.847278 ip-10-0-139-36 kubenswrapper[2575]:  > pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-h224t" podUID="f66bba6c-f7e4-4fbf-9150-a39d5fd382e6" containerName="main" containerID="cri-o://a2996b3e227481dfe81862c7663903a8d8debd3f10cd3ac5fe875d32dff68a02"
Sep 17 21:50:19.847278 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:19.847044    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-h224t" podUID="f66bba6c-f7e4-4fbf-9150-a39d5fd382e6" containerName="main" containerID="cri-o://a2996b3e227481dfe81862c7663903a8d8debd3f10cd3ac5fe875d32dff68a02" gracePeriod=60
Sep 17 21:50:19.873304 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:19.873270    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:50:20.593419 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:20.593392    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-h224t"
Sep 17 21:50:20.665936 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:20.665900    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f66bba6c-f7e4-4fbf-9150-a39d5fd382e6-home\") pod \"f66bba6c-f7e4-4fbf-9150-a39d5fd382e6\" (UID: \"f66bba6c-f7e4-4fbf-9150-a39d5fd382e6\") "
Sep 17 21:50:20.666125 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:20.665946    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f66bba6c-f7e4-4fbf-9150-a39d5fd382e6-model-cache\") pod \"f66bba6c-f7e4-4fbf-9150-a39d5fd382e6\" (UID: \"f66bba6c-f7e4-4fbf-9150-a39d5fd382e6\") "
Sep 17 21:50:20.666125 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:20.665987    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f66bba6c-f7e4-4fbf-9150-a39d5fd382e6-tmp-dir\") pod \"f66bba6c-f7e4-4fbf-9150-a39d5fd382e6\" (UID: \"f66bba6c-f7e4-4fbf-9150-a39d5fd382e6\") "
Sep 17 21:50:20.666125 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:20.666008    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x2d2v\" (UniqueName: \"kubernetes.io/projected/f66bba6c-f7e4-4fbf-9150-a39d5fd382e6-kube-api-access-x2d2v\") pod \"f66bba6c-f7e4-4fbf-9150-a39d5fd382e6\" (UID: \"f66bba6c-f7e4-4fbf-9150-a39d5fd382e6\") "
Sep 17 21:50:20.666125 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:20.666035    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f66bba6c-f7e4-4fbf-9150-a39d5fd382e6-tls-certs\") pod \"f66bba6c-f7e4-4fbf-9150-a39d5fd382e6\" (UID: \"f66bba6c-f7e4-4fbf-9150-a39d5fd382e6\") "
Sep 17 21:50:20.666125 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:20.666060    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f66bba6c-f7e4-4fbf-9150-a39d5fd382e6-dshm\") pod \"f66bba6c-f7e4-4fbf-9150-a39d5fd382e6\" (UID: \"f66bba6c-f7e4-4fbf-9150-a39d5fd382e6\") "
Sep 17 21:50:20.666395 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:20.666198    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f66bba6c-f7e4-4fbf-9150-a39d5fd382e6-home" (OuterVolumeSpecName: "home") pod "f66bba6c-f7e4-4fbf-9150-a39d5fd382e6" (UID: "f66bba6c-f7e4-4fbf-9150-a39d5fd382e6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:50:20.666395 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:20.666300    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f66bba6c-f7e4-4fbf-9150-a39d5fd382e6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f66bba6c-f7e4-4fbf-9150-a39d5fd382e6" (UID: "f66bba6c-f7e4-4fbf-9150-a39d5fd382e6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:50:20.666395 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:20.666314    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f66bba6c-f7e4-4fbf-9150-a39d5fd382e6-model-cache" (OuterVolumeSpecName: "model-cache") pod "f66bba6c-f7e4-4fbf-9150-a39d5fd382e6" (UID: "f66bba6c-f7e4-4fbf-9150-a39d5fd382e6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:50:20.666395 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:20.666362    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f66bba6c-f7e4-4fbf-9150-a39d5fd382e6-home\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:50:20.668352 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:20.668322    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f66bba6c-f7e4-4fbf-9150-a39d5fd382e6-dshm" (OuterVolumeSpecName: "dshm") pod "f66bba6c-f7e4-4fbf-9150-a39d5fd382e6" (UID: "f66bba6c-f7e4-4fbf-9150-a39d5fd382e6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:50:20.668478 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:20.668390    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f66bba6c-f7e4-4fbf-9150-a39d5fd382e6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f66bba6c-f7e4-4fbf-9150-a39d5fd382e6" (UID: "f66bba6c-f7e4-4fbf-9150-a39d5fd382e6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:50:20.668478 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:20.668403    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f66bba6c-f7e4-4fbf-9150-a39d5fd382e6-kube-api-access-x2d2v" (OuterVolumeSpecName: "kube-api-access-x2d2v") pod "f66bba6c-f7e4-4fbf-9150-a39d5fd382e6" (UID: "f66bba6c-f7e4-4fbf-9150-a39d5fd382e6"). InnerVolumeSpecName "kube-api-access-x2d2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:50:20.767083 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:20.766993    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f66bba6c-f7e4-4fbf-9150-a39d5fd382e6-model-cache\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:50:20.767083 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:20.767022    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f66bba6c-f7e4-4fbf-9150-a39d5fd382e6-tmp-dir\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:50:20.767083 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:20.767032    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2d2v\" (UniqueName: \"kubernetes.io/projected/f66bba6c-f7e4-4fbf-9150-a39d5fd382e6-kube-api-access-x2d2v\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:50:20.767083 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:20.767042    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f66bba6c-f7e4-4fbf-9150-a39d5fd382e6-tls-certs\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:50:20.767083 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:20.767052    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f66bba6c-f7e4-4fbf-9150-a39d5fd382e6-dshm\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:50:20.789399 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:20.789363    2575 generic.go:356] "Generic (PLEG): container finished" podID="f66bba6c-f7e4-4fbf-9150-a39d5fd382e6" containerID="a2996b3e227481dfe81862c7663903a8d8debd3f10cd3ac5fe875d32dff68a02" exitCode=0
Sep 17 21:50:20.789573 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:20.789416    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-h224t" event={"ID":"f66bba6c-f7e4-4fbf-9150-a39d5fd382e6","Type":"ContainerDied","Data":"a2996b3e227481dfe81862c7663903a8d8debd3f10cd3ac5fe875d32dff68a02"}
Sep 17 21:50:20.789573 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:20.789455    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-h224t" event={"ID":"f66bba6c-f7e4-4fbf-9150-a39d5fd382e6","Type":"ContainerDied","Data":"dbcb9f3afefdd2e202c97b4771342dc1a4af496ce0cef9d483705430a06bf44f"}
Sep 17 21:50:20.789573 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:20.789479    2575 scope.go:117] "RemoveContainer" containerID="a2996b3e227481dfe81862c7663903a8d8debd3f10cd3ac5fe875d32dff68a02"
Sep 17 21:50:20.789573 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:20.789416    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-h224t"
Sep 17 21:50:20.798793 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:20.798773    2575 scope.go:117] "RemoveContainer" containerID="a2996b3e227481dfe81862c7663903a8d8debd3f10cd3ac5fe875d32dff68a02"
Sep 17 21:50:20.799109 ip-10-0-139-36 kubenswrapper[2575]: E0917 21:50:20.799088    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2996b3e227481dfe81862c7663903a8d8debd3f10cd3ac5fe875d32dff68a02\": container with ID starting with a2996b3e227481dfe81862c7663903a8d8debd3f10cd3ac5fe875d32dff68a02 not found: ID does not exist" containerID="a2996b3e227481dfe81862c7663903a8d8debd3f10cd3ac5fe875d32dff68a02"
Sep 17 21:50:20.799365 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:20.799117    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2996b3e227481dfe81862c7663903a8d8debd3f10cd3ac5fe875d32dff68a02"} err="failed to get container status \"a2996b3e227481dfe81862c7663903a8d8debd3f10cd3ac5fe875d32dff68a02\": rpc error: code = NotFound desc = could not find container \"a2996b3e227481dfe81862c7663903a8d8debd3f10cd3ac5fe875d32dff68a02\": container with ID starting with a2996b3e227481dfe81862c7663903a8d8debd3f10cd3ac5fe875d32dff68a02 not found: ID does not exist"
Sep 17 21:50:20.813257 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:20.813229    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-h224t"]
Sep 17 21:50:20.815007 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:20.814983    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-h224t"]
Sep 17 21:50:21.356785 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:21.356753    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f66bba6c-f7e4-4fbf-9150-a39d5fd382e6" path="/var/lib/kubelet/pods/f66bba6c-f7e4-4fbf-9150-a39d5fd382e6/volumes"
Sep 17 21:50:22.267170 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:22.267133    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:50:22.372833 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:22.372791    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:50:28.788176 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:28.788135    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:50:28.887004 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:28.886951    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:50:29.081162 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:29.081077    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:50:33.905044 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:33.905011    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-74bbc7786d-sbpbh"]
Sep 17 21:50:33.905447 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:33.905431    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f66bba6c-f7e4-4fbf-9150-a39d5fd382e6" containerName="main"
Sep 17 21:50:33.905498 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:33.905449    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f66bba6c-f7e4-4fbf-9150-a39d5fd382e6" containerName="main"
Sep 17 21:50:33.905534 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:33.905512    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f66bba6c-f7e4-4fbf-9150-a39d5fd382e6" containerName="main"
Sep 17 21:50:33.909672 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:33.909654    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-74bbc7786d-sbpbh"
Sep 17 21:50:33.912195 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:33.912168    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 21:50:33.912335 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:33.912197    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Sep 17 21:50:33.912550 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:33.912533    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"seaweedfs-s3-creds\""
Sep 17 21:50:33.914641 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:33.914623    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Sep 17 21:50:33.914903 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:33.914885    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Sep 17 21:50:33.915020 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:33.915001    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-pn4lg\""
Sep 17 21:50:33.929151 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:33.929128    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-74bbc7786d-sbpbh"]
Sep 17 21:50:33.945762 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:33.945737    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:50:34.083944 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:34.083908    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92fac342-efb6-484b-bd71-f6e9cb2e94fd-home\") pod \"tls-verification-test-kserve-74bbc7786d-sbpbh\" (UID: \"92fac342-efb6-484b-bd71-f6e9cb2e94fd\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-74bbc7786d-sbpbh"
Sep 17 21:50:34.083944 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:34.083945    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92fac342-efb6-484b-bd71-f6e9cb2e94fd-tmp-dir\") pod \"tls-verification-test-kserve-74bbc7786d-sbpbh\" (UID: \"92fac342-efb6-484b-bd71-f6e9cb2e94fd\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-74bbc7786d-sbpbh"
Sep 17 21:50:34.084231 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:34.083963    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92fac342-efb6-484b-bd71-f6e9cb2e94fd-tls-certs\") pod \"tls-verification-test-kserve-74bbc7786d-sbpbh\" (UID: \"92fac342-efb6-484b-bd71-f6e9cb2e94fd\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-74bbc7786d-sbpbh"
Sep 17 21:50:34.084231 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:34.083994    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92fac342-efb6-484b-bd71-f6e9cb2e94fd-kserve-provision-location\") pod \"tls-verification-test-kserve-74bbc7786d-sbpbh\" (UID: \"92fac342-efb6-484b-bd71-f6e9cb2e94fd\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-74bbc7786d-sbpbh"
Sep 17 21:50:34.084231 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:34.084093    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfz5n\" (UniqueName: \"kubernetes.io/projected/92fac342-efb6-484b-bd71-f6e9cb2e94fd-kube-api-access-cfz5n\") pod \"tls-verification-test-kserve-74bbc7786d-sbpbh\" (UID: \"92fac342-efb6-484b-bd71-f6e9cb2e94fd\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-74bbc7786d-sbpbh"
Sep 17 21:50:34.084231 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:34.084152    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/92fac342-efb6-484b-bd71-f6e9cb2e94fd-cabundle-cert\") pod \"tls-verification-test-kserve-74bbc7786d-sbpbh\" (UID: \"92fac342-efb6-484b-bd71-f6e9cb2e94fd\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-74bbc7786d-sbpbh"
Sep 17 21:50:34.084231 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:34.084220    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92fac342-efb6-484b-bd71-f6e9cb2e94fd-model-cache\") pod \"tls-verification-test-kserve-74bbc7786d-sbpbh\" (UID: \"92fac342-efb6-484b-bd71-f6e9cb2e94fd\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-74bbc7786d-sbpbh"
Sep 17 21:50:34.084435 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:34.084254    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92fac342-efb6-484b-bd71-f6e9cb2e94fd-dshm\") pod \"tls-verification-test-kserve-74bbc7786d-sbpbh\" (UID: \"92fac342-efb6-484b-bd71-f6e9cb2e94fd\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-74bbc7786d-sbpbh"
Sep 17 21:50:34.170174 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:34.170074    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:50:34.185121 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:34.185086    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92fac342-efb6-484b-bd71-f6e9cb2e94fd-dshm\") pod \"tls-verification-test-kserve-74bbc7786d-sbpbh\" (UID: \"92fac342-efb6-484b-bd71-f6e9cb2e94fd\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-74bbc7786d-sbpbh"
Sep 17 21:50:34.185301 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:34.185139    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92fac342-efb6-484b-bd71-f6e9cb2e94fd-home\") pod \"tls-verification-test-kserve-74bbc7786d-sbpbh\" (UID: \"92fac342-efb6-484b-bd71-f6e9cb2e94fd\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-74bbc7786d-sbpbh"
Sep 17 21:50:34.185301 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:34.185155    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92fac342-efb6-484b-bd71-f6e9cb2e94fd-tmp-dir\") pod \"tls-verification-test-kserve-74bbc7786d-sbpbh\" (UID: \"92fac342-efb6-484b-bd71-f6e9cb2e94fd\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-74bbc7786d-sbpbh"
Sep 17 21:50:34.185301 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:34.185169    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92fac342-efb6-484b-bd71-f6e9cb2e94fd-tls-certs\") pod \"tls-verification-test-kserve-74bbc7786d-sbpbh\" (UID: \"92fac342-efb6-484b-bd71-f6e9cb2e94fd\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-74bbc7786d-sbpbh"
Sep 17 21:50:34.185301 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:34.185207    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92fac342-efb6-484b-bd71-f6e9cb2e94fd-kserve-provision-location\") pod \"tls-verification-test-kserve-74bbc7786d-sbpbh\" (UID: \"92fac342-efb6-484b-bd71-f6e9cb2e94fd\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-74bbc7786d-sbpbh"
Sep 17 21:50:34.185301 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:34.185230    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfz5n\" (UniqueName: \"kubernetes.io/projected/92fac342-efb6-484b-bd71-f6e9cb2e94fd-kube-api-access-cfz5n\") pod \"tls-verification-test-kserve-74bbc7786d-sbpbh\" (UID: \"92fac342-efb6-484b-bd71-f6e9cb2e94fd\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-74bbc7786d-sbpbh"
Sep 17 21:50:34.185569 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:34.185390    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/92fac342-efb6-484b-bd71-f6e9cb2e94fd-cabundle-cert\") pod \"tls-verification-test-kserve-74bbc7786d-sbpbh\" (UID: \"92fac342-efb6-484b-bd71-f6e9cb2e94fd\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-74bbc7786d-sbpbh"
Sep 17 21:50:34.185569 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:34.185464    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92fac342-efb6-484b-bd71-f6e9cb2e94fd-model-cache\") pod \"tls-verification-test-kserve-74bbc7786d-sbpbh\" (UID: \"92fac342-efb6-484b-bd71-f6e9cb2e94fd\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-74bbc7786d-sbpbh"
Sep 17 21:50:34.185685 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:34.185615    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92fac342-efb6-484b-bd71-f6e9cb2e94fd-tmp-dir\") pod \"tls-verification-test-kserve-74bbc7786d-sbpbh\" (UID: \"92fac342-efb6-484b-bd71-f6e9cb2e94fd\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-74bbc7786d-sbpbh"
Sep 17 21:50:34.185685 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:34.185664    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92fac342-efb6-484b-bd71-f6e9cb2e94fd-home\") pod \"tls-verification-test-kserve-74bbc7786d-sbpbh\" (UID: \"92fac342-efb6-484b-bd71-f6e9cb2e94fd\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-74bbc7786d-sbpbh"
Sep 17 21:50:34.185886 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:34.185863    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92fac342-efb6-484b-bd71-f6e9cb2e94fd-model-cache\") pod \"tls-verification-test-kserve-74bbc7786d-sbpbh\" (UID: \"92fac342-efb6-484b-bd71-f6e9cb2e94fd\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-74bbc7786d-sbpbh"
Sep 17 21:50:34.185958 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:34.185895    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92fac342-efb6-484b-bd71-f6e9cb2e94fd-kserve-provision-location\") pod \"tls-verification-test-kserve-74bbc7786d-sbpbh\" (UID: \"92fac342-efb6-484b-bd71-f6e9cb2e94fd\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-74bbc7786d-sbpbh"
Sep 17 21:50:34.186149 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:34.186131    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/92fac342-efb6-484b-bd71-f6e9cb2e94fd-cabundle-cert\") pod \"tls-verification-test-kserve-74bbc7786d-sbpbh\" (UID: \"92fac342-efb6-484b-bd71-f6e9cb2e94fd\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-74bbc7786d-sbpbh"
Sep 17 21:50:34.188134 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:34.188109    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92fac342-efb6-484b-bd71-f6e9cb2e94fd-dshm\") pod \"tls-verification-test-kserve-74bbc7786d-sbpbh\" (UID: \"92fac342-efb6-484b-bd71-f6e9cb2e94fd\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-74bbc7786d-sbpbh"
Sep 17 21:50:34.188276 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:34.188259    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92fac342-efb6-484b-bd71-f6e9cb2e94fd-tls-certs\") pod \"tls-verification-test-kserve-74bbc7786d-sbpbh\" (UID: \"92fac342-efb6-484b-bd71-f6e9cb2e94fd\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-74bbc7786d-sbpbh"
Sep 17 21:50:34.192955 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:34.192936    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfz5n\" (UniqueName: \"kubernetes.io/projected/92fac342-efb6-484b-bd71-f6e9cb2e94fd-kube-api-access-cfz5n\") pod \"tls-verification-test-kserve-74bbc7786d-sbpbh\" (UID: \"92fac342-efb6-484b-bd71-f6e9cb2e94fd\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-74bbc7786d-sbpbh"
Sep 17 21:50:34.220978 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:34.220952    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-74bbc7786d-sbpbh"
Sep 17 21:50:34.514764 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:34.514735    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-74bbc7786d-sbpbh"]
Sep 17 21:50:34.517648 ip-10-0-139-36 kubenswrapper[2575]: W0917 21:50:34.517603    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92fac342_efb6_484b_bd71_f6e9cb2e94fd.slice/crio-4170ea3a5d2075a15752f0266ce605c6057037b6031b4bee904dbfe64946aa3a WatchSource:0}: Error finding container 4170ea3a5d2075a15752f0266ce605c6057037b6031b4bee904dbfe64946aa3a: Status 404 returned error can't find the container with id 4170ea3a5d2075a15752f0266ce605c6057037b6031b4bee904dbfe64946aa3a
Sep 17 21:50:34.861266 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:34.861169    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-74bbc7786d-sbpbh" event={"ID":"92fac342-efb6-484b-bd71-f6e9cb2e94fd","Type":"ContainerStarted","Data":"b78a597ca9e8890982d0f97044284e91d830a386dd0dd7e65e33678430cd1458"}
Sep 17 21:50:34.861266 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:34.861237    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-74bbc7786d-sbpbh" event={"ID":"92fac342-efb6-484b-bd71-f6e9cb2e94fd","Type":"ContainerStarted","Data":"4170ea3a5d2075a15752f0266ce605c6057037b6031b4bee904dbfe64946aa3a"}
Sep 17 21:50:35.997951 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:35.997916    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:50:36.096806 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:36.096763    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:50:36.276246 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:36.276156    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:50:36.344463 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:36.344417    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:50:36.872704 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:36.872672    2575 generic.go:356] "Generic (PLEG): container finished" podID="92fac342-efb6-484b-bd71-f6e9cb2e94fd" containerID="b78a597ca9e8890982d0f97044284e91d830a386dd0dd7e65e33678430cd1458" exitCode=0
Sep 17 21:50:36.872903 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:36.872749    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-74bbc7786d-sbpbh" event={"ID":"92fac342-efb6-484b-bd71-f6e9cb2e94fd","Type":"ContainerDied","Data":"b78a597ca9e8890982d0f97044284e91d830a386dd0dd7e65e33678430cd1458"}
Sep 17 21:50:37.880231 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:37.880193    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-74bbc7786d-sbpbh" event={"ID":"92fac342-efb6-484b-bd71-f6e9cb2e94fd","Type":"ContainerStarted","Data":"19f28389b9c1b8c8ddb1e2978f45ec484dd00e4ca5a29a47fe213f987f6d46c0"}
Sep 17 21:50:37.912503 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:37.912440    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-74bbc7786d-sbpbh" podStartSLOduration=2.7307655779999997 podStartE2EDuration="4.912424429s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.181658851s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:50:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:50:37.90990714 +0000 UTC m=+3703.062351516" watchObservedRunningTime="2026-09-17 21:50:37.912424429 +0000 UTC m=+3703.064868804"
Sep 17 21:50:38.221596 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:38.221558    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-74bbc7786d-sbpbh"
Sep 17 21:50:44.222145 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:44.222112    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-74bbc7786d-sbpbh"
Sep 17 21:50:44.223948 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:44.223920    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-74bbc7786d-sbpbh" podUID="92fac342-efb6-484b-bd71-f6e9cb2e94fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Sep 17 21:50:48.663529 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:48.663474    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:50:54.222027 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:50:54.221982    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-74bbc7786d-sbpbh" podUID="92fac342-efb6-484b-bd71-f6e9cb2e94fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Sep 17 21:51:04.222189 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:51:04.222142    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-74bbc7786d-sbpbh" podUID="92fac342-efb6-484b-bd71-f6e9cb2e94fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Sep 17 21:51:09.771007 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:51:09.770961    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:51:09.864571 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:51:09.864531    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:51:14.221900 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:51:14.221838    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-74bbc7786d-sbpbh" podUID="92fac342-efb6-484b-bd71-f6e9cb2e94fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Sep 17 21:51:15.629653 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:51:15.629615    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:51:15.954126 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:51:15.954091    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:51:16.764895 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:51:16.764825    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:51:23.328268 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:51:23.328223    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:51:23.460187 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:51:23.460147    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:51:24.221523 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:51:24.221474    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-74bbc7786d-sbpbh" podUID="92fac342-efb6-484b-bd71-f6e9cb2e94fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Sep 17 21:51:34.222309 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:51:34.222261    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-74bbc7786d-sbpbh" podUID="92fac342-efb6-484b-bd71-f6e9cb2e94fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Sep 17 21:51:42.229795 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:51:42.229748    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:51:44.221870 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:51:44.221807    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-74bbc7786d-sbpbh" podUID="92fac342-efb6-484b-bd71-f6e9cb2e94fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Sep 17 21:51:49.045227 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:51:49.045109    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:51:49.144888 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:51:49.144830    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:51:54.221590 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:51:54.221549    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-74bbc7786d-sbpbh" podUID="92fac342-efb6-484b-bd71-f6e9cb2e94fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Sep 17 21:52:04.221447 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:52:04.221393    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-74bbc7786d-sbpbh" podUID="92fac342-efb6-484b-bd71-f6e9cb2e94fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Sep 17 21:52:14.222246 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:52:14.222201    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-74bbc7786d-sbpbh" podUID="92fac342-efb6-484b-bd71-f6e9cb2e94fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Sep 17 21:52:24.222301 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:52:24.222260    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-74bbc7786d-sbpbh" podUID="92fac342-efb6-484b-bd71-f6e9cb2e94fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Sep 17 21:52:34.231813 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:52:34.231782    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-74bbc7786d-sbpbh"
Sep 17 21:52:34.239483 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:52:34.239459    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-74bbc7786d-sbpbh"
Sep 17 21:52:38.707884 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:52:38.707789    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:52:39.464295 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:52:39.464258    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-74bbc7786d-sbpbh"]
Sep 17 21:52:40.092007 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:52:40.091969    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:52:43.751775 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:52:43.751733    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:52:43.991975 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:52:43.991939    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:52:45.868359 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:52:45.868326    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:52:50.296322 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:52:50.296288    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:52:52.759901 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:52:52.759835    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:52:52.842874 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:52:52.842816    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:52:54.486892 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:52:54.486812    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-74bbc7786d-sbpbh" podUID="92fac342-efb6-484b-bd71-f6e9cb2e94fd" containerName="main" containerID="cri-o://19f28389b9c1b8c8ddb1e2978f45ec484dd00e4ca5a29a47fe213f987f6d46c0" gracePeriod=45
Sep 17 21:52:55.234643 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:52:55.234619    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-74bbc7786d-sbpbh"
Sep 17 21:52:55.276720 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:52:55.276658    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/92fac342-efb6-484b-bd71-f6e9cb2e94fd-cabundle-cert\") pod \"92fac342-efb6-484b-bd71-f6e9cb2e94fd\" (UID: \"92fac342-efb6-484b-bd71-f6e9cb2e94fd\") "
Sep 17 21:52:55.276720 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:52:55.276712    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92fac342-efb6-484b-bd71-f6e9cb2e94fd-home\") pod \"92fac342-efb6-484b-bd71-f6e9cb2e94fd\" (UID: \"92fac342-efb6-484b-bd71-f6e9cb2e94fd\") "
Sep 17 21:52:55.276956 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:52:55.276737    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92fac342-efb6-484b-bd71-f6e9cb2e94fd-tls-certs\") pod \"92fac342-efb6-484b-bd71-f6e9cb2e94fd\" (UID: \"92fac342-efb6-484b-bd71-f6e9cb2e94fd\") "
Sep 17 21:52:55.276956 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:52:55.276815    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cfz5n\" (UniqueName: \"kubernetes.io/projected/92fac342-efb6-484b-bd71-f6e9cb2e94fd-kube-api-access-cfz5n\") pod \"92fac342-efb6-484b-bd71-f6e9cb2e94fd\" (UID: \"92fac342-efb6-484b-bd71-f6e9cb2e94fd\") "
Sep 17 21:52:55.276956 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:52:55.276844    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92fac342-efb6-484b-bd71-f6e9cb2e94fd-dshm\") pod \"92fac342-efb6-484b-bd71-f6e9cb2e94fd\" (UID: \"92fac342-efb6-484b-bd71-f6e9cb2e94fd\") "
Sep 17 21:52:55.276956 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:52:55.276905    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92fac342-efb6-484b-bd71-f6e9cb2e94fd-kserve-provision-location\") pod \"92fac342-efb6-484b-bd71-f6e9cb2e94fd\" (UID: \"92fac342-efb6-484b-bd71-f6e9cb2e94fd\") "
Sep 17 21:52:55.277189 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:52:55.276963    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92fac342-efb6-484b-bd71-f6e9cb2e94fd-tmp-dir\") pod \"92fac342-efb6-484b-bd71-f6e9cb2e94fd\" (UID: \"92fac342-efb6-484b-bd71-f6e9cb2e94fd\") "
Sep 17 21:52:55.277189 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:52:55.276988    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92fac342-efb6-484b-bd71-f6e9cb2e94fd-model-cache\") pod \"92fac342-efb6-484b-bd71-f6e9cb2e94fd\" (UID: \"92fac342-efb6-484b-bd71-f6e9cb2e94fd\") "
Sep 17 21:52:55.277621 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:52:55.277450    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92fac342-efb6-484b-bd71-f6e9cb2e94fd-model-cache" (OuterVolumeSpecName: "model-cache") pod "92fac342-efb6-484b-bd71-f6e9cb2e94fd" (UID: "92fac342-efb6-484b-bd71-f6e9cb2e94fd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:52:55.277731 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:52:55.277687    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92fac342-efb6-484b-bd71-f6e9cb2e94fd-home" (OuterVolumeSpecName: "home") pod "92fac342-efb6-484b-bd71-f6e9cb2e94fd" (UID: "92fac342-efb6-484b-bd71-f6e9cb2e94fd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:52:55.277935 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:52:55.277888    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92fac342-efb6-484b-bd71-f6e9cb2e94fd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "92fac342-efb6-484b-bd71-f6e9cb2e94fd" (UID: "92fac342-efb6-484b-bd71-f6e9cb2e94fd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:52:55.280187 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:52:55.280149    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92fac342-efb6-484b-bd71-f6e9cb2e94fd-dshm" (OuterVolumeSpecName: "dshm") pod "92fac342-efb6-484b-bd71-f6e9cb2e94fd" (UID: "92fac342-efb6-484b-bd71-f6e9cb2e94fd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:52:55.280287 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:52:55.280220    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92fac342-efb6-484b-bd71-f6e9cb2e94fd-kube-api-access-cfz5n" (OuterVolumeSpecName: "kube-api-access-cfz5n") pod "92fac342-efb6-484b-bd71-f6e9cb2e94fd" (UID: "92fac342-efb6-484b-bd71-f6e9cb2e94fd"). InnerVolumeSpecName "kube-api-access-cfz5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:52:55.280287 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:52:55.280220    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92fac342-efb6-484b-bd71-f6e9cb2e94fd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "92fac342-efb6-484b-bd71-f6e9cb2e94fd" (UID: "92fac342-efb6-484b-bd71-f6e9cb2e94fd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:52:55.290596 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:52:55.290570    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92fac342-efb6-484b-bd71-f6e9cb2e94fd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "92fac342-efb6-484b-bd71-f6e9cb2e94fd" (UID: "92fac342-efb6-484b-bd71-f6e9cb2e94fd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:52:55.309175 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:52:55.309149    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92fac342-efb6-484b-bd71-f6e9cb2e94fd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "92fac342-efb6-484b-bd71-f6e9cb2e94fd" (UID: "92fac342-efb6-484b-bd71-f6e9cb2e94fd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:52:55.378315 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:52:55.378288    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92fac342-efb6-484b-bd71-f6e9cb2e94fd-home\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:52:55.378428 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:52:55.378317    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92fac342-efb6-484b-bd71-f6e9cb2e94fd-tls-certs\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:52:55.378428 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:52:55.378334    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfz5n\" (UniqueName: \"kubernetes.io/projected/92fac342-efb6-484b-bd71-f6e9cb2e94fd-kube-api-access-cfz5n\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:52:55.378428 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:52:55.378347    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92fac342-efb6-484b-bd71-f6e9cb2e94fd-dshm\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:52:55.378428 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:52:55.378376    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92fac342-efb6-484b-bd71-f6e9cb2e94fd-kserve-provision-location\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:52:55.378428 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:52:55.378400    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92fac342-efb6-484b-bd71-f6e9cb2e94fd-tmp-dir\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:52:55.378428 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:52:55.378417    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92fac342-efb6-484b-bd71-f6e9cb2e94fd-model-cache\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:52:55.378428 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:52:55.378431    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/92fac342-efb6-484b-bd71-f6e9cb2e94fd-cabundle-cert\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:52:55.606685 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:52:55.606599    2575 generic.go:356] "Generic (PLEG): container finished" podID="92fac342-efb6-484b-bd71-f6e9cb2e94fd" containerID="19f28389b9c1b8c8ddb1e2978f45ec484dd00e4ca5a29a47fe213f987f6d46c0" exitCode=0
Sep 17 21:52:55.606685 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:52:55.606645    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-74bbc7786d-sbpbh" event={"ID":"92fac342-efb6-484b-bd71-f6e9cb2e94fd","Type":"ContainerDied","Data":"19f28389b9c1b8c8ddb1e2978f45ec484dd00e4ca5a29a47fe213f987f6d46c0"}
Sep 17 21:52:55.606685 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:52:55.606659    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-74bbc7786d-sbpbh"
Sep 17 21:52:55.607232 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:52:55.606700    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-74bbc7786d-sbpbh" event={"ID":"92fac342-efb6-484b-bd71-f6e9cb2e94fd","Type":"ContainerDied","Data":"4170ea3a5d2075a15752f0266ce605c6057037b6031b4bee904dbfe64946aa3a"}
Sep 17 21:52:55.607232 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:52:55.606735    2575 scope.go:117] "RemoveContainer" containerID="19f28389b9c1b8c8ddb1e2978f45ec484dd00e4ca5a29a47fe213f987f6d46c0"
Sep 17 21:52:55.615148 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:52:55.615127    2575 scope.go:117] "RemoveContainer" containerID="b78a597ca9e8890982d0f97044284e91d830a386dd0dd7e65e33678430cd1458"
Sep 17 21:52:55.624742 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:52:55.624717    2575 scope.go:117] "RemoveContainer" containerID="19f28389b9c1b8c8ddb1e2978f45ec484dd00e4ca5a29a47fe213f987f6d46c0"
Sep 17 21:52:55.625664 ip-10-0-139-36 kubenswrapper[2575]: E0917 21:52:55.625427    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19f28389b9c1b8c8ddb1e2978f45ec484dd00e4ca5a29a47fe213f987f6d46c0\": container with ID starting with 19f28389b9c1b8c8ddb1e2978f45ec484dd00e4ca5a29a47fe213f987f6d46c0 not found: ID does not exist" containerID="19f28389b9c1b8c8ddb1e2978f45ec484dd00e4ca5a29a47fe213f987f6d46c0"
Sep 17 21:52:55.625664 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:52:55.625465    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19f28389b9c1b8c8ddb1e2978f45ec484dd00e4ca5a29a47fe213f987f6d46c0"} err="failed to get container status \"19f28389b9c1b8c8ddb1e2978f45ec484dd00e4ca5a29a47fe213f987f6d46c0\": rpc error: code = NotFound desc = could not find container \"19f28389b9c1b8c8ddb1e2978f45ec484dd00e4ca5a29a47fe213f987f6d46c0\": container with ID starting with 19f28389b9c1b8c8ddb1e2978f45ec484dd00e4ca5a29a47fe213f987f6d46c0 not found: ID does not exist"
Sep 17 21:52:55.625664 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:52:55.625489    2575 scope.go:117] "RemoveContainer" containerID="b78a597ca9e8890982d0f97044284e91d830a386dd0dd7e65e33678430cd1458"
Sep 17 21:52:55.626028 ip-10-0-139-36 kubenswrapper[2575]: E0917 21:52:55.625998    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b78a597ca9e8890982d0f97044284e91d830a386dd0dd7e65e33678430cd1458\": container with ID starting with b78a597ca9e8890982d0f97044284e91d830a386dd0dd7e65e33678430cd1458 not found: ID does not exist" containerID="b78a597ca9e8890982d0f97044284e91d830a386dd0dd7e65e33678430cd1458"
Sep 17 21:52:55.626114 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:52:55.626033    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b78a597ca9e8890982d0f97044284e91d830a386dd0dd7e65e33678430cd1458"} err="failed to get container status \"b78a597ca9e8890982d0f97044284e91d830a386dd0dd7e65e33678430cd1458\": rpc error: code = NotFound desc = could not find container \"b78a597ca9e8890982d0f97044284e91d830a386dd0dd7e65e33678430cd1458\": container with ID starting with b78a597ca9e8890982d0f97044284e91d830a386dd0dd7e65e33678430cd1458 not found: ID does not exist"
Sep 17 21:52:55.627921 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:52:55.627898    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-74bbc7786d-sbpbh"]
Sep 17 21:52:55.633318 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:52:55.633292    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-74bbc7786d-sbpbh"]
Sep 17 21:52:57.357011 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:52:57.356978    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92fac342-efb6-484b-bd71-f6e9cb2e94fd" path="/var/lib/kubelet/pods/92fac342-efb6-484b-bd71-f6e9cb2e94fd/volumes"
Sep 17 21:53:17.749246 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:53:17.749206    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:53:17.847791 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:53:17.847702    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:53:29.158557 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:53:29.158516    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:53:29.250448 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:53:29.250407    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:53:35.142687 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:53:35.142649    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:53:35.407523 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:53:35.407426    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:53:37.470592 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:53:37.470549    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:53:43.152187 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:53:43.152152    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:53:43.210246 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:53:43.210217    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:53:55.759255 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:53:55.759219    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:53:55.860135 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:53:55.860100    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:53:57.827339 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:53:57.827299    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:54:06.467004 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:06.466966    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:54:07.669227 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:07.669138    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:54:07.764578 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:07.764546    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:54:14.233277 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:14.233240    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:54:20.193157 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:20.193123    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:54:20.259018 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:20.258986    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:54:21.599316 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:21.599282    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h9p96"]
Sep 17 21:54:21.599789 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:21.599759    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92fac342-efb6-484b-bd71-f6e9cb2e94fd" containerName="main"
Sep 17 21:54:21.599789 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:21.599786    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="92fac342-efb6-484b-bd71-f6e9cb2e94fd" containerName="main"
Sep 17 21:54:21.599976 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:21.599820    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92fac342-efb6-484b-bd71-f6e9cb2e94fd" containerName="storage-initializer"
Sep 17 21:54:21.599976 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:21.599829    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="92fac342-efb6-484b-bd71-f6e9cb2e94fd" containerName="storage-initializer"
Sep 17 21:54:21.599976 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:21.599943    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="92fac342-efb6-484b-bd71-f6e9cb2e94fd" containerName="main"
Sep 17 21:54:21.602863 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:21.602833    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h9p96"
Sep 17 21:54:21.605468 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:21.605232    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Sep 17 21:54:21.606047 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:21.606022    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Sep 17 21:54:21.606185 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:21.606063    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Sep 17 21:54:21.606185 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:21.606065    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-22r9q\""
Sep 17 21:54:21.613892 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:21.613839    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h9p96"]
Sep 17 21:54:21.628983 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:21.628957    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:54:21.748897 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:21.748827    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d7fc31e-6876-43a5-8d90-7269f9f79d56-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-h9p96\" (UID: \"0d7fc31e-6876-43a5-8d90-7269f9f79d56\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h9p96"
Sep 17 21:54:21.749074 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:21.748920    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d7fc31e-6876-43a5-8d90-7269f9f79d56-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-h9p96\" (UID: \"0d7fc31e-6876-43a5-8d90-7269f9f79d56\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h9p96"
Sep 17 21:54:21.749074 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:21.748976    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d7fc31e-6876-43a5-8d90-7269f9f79d56-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-h9p96\" (UID: \"0d7fc31e-6876-43a5-8d90-7269f9f79d56\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h9p96"
Sep 17 21:54:21.749074 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:21.749008    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d7fc31e-6876-43a5-8d90-7269f9f79d56-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-h9p96\" (UID: \"0d7fc31e-6876-43a5-8d90-7269f9f79d56\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h9p96"
Sep 17 21:54:21.749074 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:21.749026    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d7fc31e-6876-43a5-8d90-7269f9f79d56-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-h9p96\" (UID: \"0d7fc31e-6876-43a5-8d90-7269f9f79d56\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h9p96"
Sep 17 21:54:21.749074 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:21.749045    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dsss\" (UniqueName: \"kubernetes.io/projected/0d7fc31e-6876-43a5-8d90-7269f9f79d56-kube-api-access-8dsss\") pod \"rolling-upgrade-test-kserve-74b79d7f77-h9p96\" (UID: \"0d7fc31e-6876-43a5-8d90-7269f9f79d56\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h9p96"
Sep 17 21:54:21.850361 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:21.850280    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d7fc31e-6876-43a5-8d90-7269f9f79d56-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-h9p96\" (UID: \"0d7fc31e-6876-43a5-8d90-7269f9f79d56\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h9p96"
Sep 17 21:54:21.850361 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:21.850328    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d7fc31e-6876-43a5-8d90-7269f9f79d56-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-h9p96\" (UID: \"0d7fc31e-6876-43a5-8d90-7269f9f79d56\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h9p96"
Sep 17 21:54:21.850361 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:21.850345    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d7fc31e-6876-43a5-8d90-7269f9f79d56-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-h9p96\" (UID: \"0d7fc31e-6876-43a5-8d90-7269f9f79d56\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h9p96"
Sep 17 21:54:21.850597 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:21.850365    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dsss\" (UniqueName: \"kubernetes.io/projected/0d7fc31e-6876-43a5-8d90-7269f9f79d56-kube-api-access-8dsss\") pod \"rolling-upgrade-test-kserve-74b79d7f77-h9p96\" (UID: \"0d7fc31e-6876-43a5-8d90-7269f9f79d56\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h9p96"
Sep 17 21:54:21.850597 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:21.850456    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d7fc31e-6876-43a5-8d90-7269f9f79d56-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-h9p96\" (UID: \"0d7fc31e-6876-43a5-8d90-7269f9f79d56\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h9p96"
Sep 17 21:54:21.850597 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:21.850472    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d7fc31e-6876-43a5-8d90-7269f9f79d56-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-h9p96\" (UID: \"0d7fc31e-6876-43a5-8d90-7269f9f79d56\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h9p96"
Sep 17 21:54:21.850812 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:21.850789    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d7fc31e-6876-43a5-8d90-7269f9f79d56-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-h9p96\" (UID: \"0d7fc31e-6876-43a5-8d90-7269f9f79d56\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h9p96"
Sep 17 21:54:21.850902 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:21.850837    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d7fc31e-6876-43a5-8d90-7269f9f79d56-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-h9p96\" (UID: \"0d7fc31e-6876-43a5-8d90-7269f9f79d56\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h9p96"
Sep 17 21:54:21.850994 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:21.850973    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d7fc31e-6876-43a5-8d90-7269f9f79d56-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-h9p96\" (UID: \"0d7fc31e-6876-43a5-8d90-7269f9f79d56\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h9p96"
Sep 17 21:54:21.852657 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:21.852637    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d7fc31e-6876-43a5-8d90-7269f9f79d56-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-h9p96\" (UID: \"0d7fc31e-6876-43a5-8d90-7269f9f79d56\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h9p96"
Sep 17 21:54:21.853179 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:21.853144    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d7fc31e-6876-43a5-8d90-7269f9f79d56-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-h9p96\" (UID: \"0d7fc31e-6876-43a5-8d90-7269f9f79d56\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h9p96"
Sep 17 21:54:21.858417 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:21.858388    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:54:21.859311 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:21.859292    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dsss\" (UniqueName: \"kubernetes.io/projected/0d7fc31e-6876-43a5-8d90-7269f9f79d56-kube-api-access-8dsss\") pod \"rolling-upgrade-test-kserve-74b79d7f77-h9p96\" (UID: \"0d7fc31e-6876-43a5-8d90-7269f9f79d56\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h9p96"
Sep 17 21:54:21.915143 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:21.915109    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h9p96"
Sep 17 21:54:22.193716 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:22.193630    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h9p96"]
Sep 17 21:54:22.197300 ip-10-0-139-36 kubenswrapper[2575]: W0917 21:54:22.197271    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d7fc31e_6876_43a5_8d90_7269f9f79d56.slice/crio-a8e071a3aabc49f08ea58f1317e1755abc9bc0bc60b406aa582dd0c7c15e7de8 WatchSource:0}: Error finding container a8e071a3aabc49f08ea58f1317e1755abc9bc0bc60b406aa582dd0c7c15e7de8: Status 404 returned error can't find the container with id a8e071a3aabc49f08ea58f1317e1755abc9bc0bc60b406aa582dd0c7c15e7de8
Sep 17 21:54:23.031542 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:23.031505    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h9p96" event={"ID":"0d7fc31e-6876-43a5-8d90-7269f9f79d56","Type":"ContainerStarted","Data":"24e51c518f46f187ca23f7abaa1611ed6d1227ac5a1c65bfc94401fb24343d8b"}
Sep 17 21:54:23.031953 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:23.031552    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h9p96" event={"ID":"0d7fc31e-6876-43a5-8d90-7269f9f79d56","Type":"ContainerStarted","Data":"a8e071a3aabc49f08ea58f1317e1755abc9bc0bc60b406aa582dd0c7c15e7de8"}
Sep 17 21:54:23.050152 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:23.050107    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h9p96" podStartSLOduration=2.050089427 podStartE2EDuration="2.050089427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:54:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:54:23.047963964 +0000 UTC m=+3928.200408355" watchObservedRunningTime="2026-09-17 21:54:23.050089427 +0000 UTC m=+3928.202533802"
Sep 17 21:54:23.916193 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:23.916153    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h9p96"
Sep 17 21:54:31.916002 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:31.915964    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h9p96"
Sep 17 21:54:31.928473 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:31.928432    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h9p96"
Sep 17 21:54:32.088820 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:32.088790    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h9p96"
Sep 17 21:54:45.816741 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:45.816705    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:54:55.007336 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:55.007301    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-97q4z"]
Sep 17 21:54:55.010689 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:55.010670    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-97q4z"
Sep 17 21:54:55.020578 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:55.020553    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-97q4z"]
Sep 17 21:54:55.130975 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:55.130938    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9frpf\" (UniqueName: \"kubernetes.io/projected/7fb926d1-1106-4438-bf30-3e6f37961b7c-kube-api-access-9frpf\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-97q4z\" (UID: \"7fb926d1-1106-4438-bf30-3e6f37961b7c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-97q4z"
Sep 17 21:54:55.131161 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:55.130995    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fb926d1-1106-4438-bf30-3e6f37961b7c-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-97q4z\" (UID: \"7fb926d1-1106-4438-bf30-3e6f37961b7c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-97q4z"
Sep 17 21:54:55.131161 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:55.131037    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fb926d1-1106-4438-bf30-3e6f37961b7c-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-97q4z\" (UID: \"7fb926d1-1106-4438-bf30-3e6f37961b7c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-97q4z"
Sep 17 21:54:55.131161 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:55.131064    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fb926d1-1106-4438-bf30-3e6f37961b7c-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-97q4z\" (UID: \"7fb926d1-1106-4438-bf30-3e6f37961b7c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-97q4z"
Sep 17 21:54:55.131161 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:55.131117    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fb926d1-1106-4438-bf30-3e6f37961b7c-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-97q4z\" (UID: \"7fb926d1-1106-4438-bf30-3e6f37961b7c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-97q4z"
Sep 17 21:54:55.131161 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:55.131157    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fb926d1-1106-4438-bf30-3e6f37961b7c-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-97q4z\" (UID: \"7fb926d1-1106-4438-bf30-3e6f37961b7c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-97q4z"
Sep 17 21:54:55.232003 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:55.231965    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fb926d1-1106-4438-bf30-3e6f37961b7c-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-97q4z\" (UID: \"7fb926d1-1106-4438-bf30-3e6f37961b7c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-97q4z"
Sep 17 21:54:55.232177 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:55.232031    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fb926d1-1106-4438-bf30-3e6f37961b7c-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-97q4z\" (UID: \"7fb926d1-1106-4438-bf30-3e6f37961b7c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-97q4z"
Sep 17 21:54:55.232177 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:55.232084    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9frpf\" (UniqueName: \"kubernetes.io/projected/7fb926d1-1106-4438-bf30-3e6f37961b7c-kube-api-access-9frpf\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-97q4z\" (UID: \"7fb926d1-1106-4438-bf30-3e6f37961b7c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-97q4z"
Sep 17 21:54:55.232177 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:55.232131    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fb926d1-1106-4438-bf30-3e6f37961b7c-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-97q4z\" (UID: \"7fb926d1-1106-4438-bf30-3e6f37961b7c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-97q4z"
Sep 17 21:54:55.232177 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:55.232159    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fb926d1-1106-4438-bf30-3e6f37961b7c-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-97q4z\" (UID: \"7fb926d1-1106-4438-bf30-3e6f37961b7c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-97q4z"
Sep 17 21:54:55.232383 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:55.232186    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fb926d1-1106-4438-bf30-3e6f37961b7c-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-97q4z\" (UID: \"7fb926d1-1106-4438-bf30-3e6f37961b7c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-97q4z"
Sep 17 21:54:55.232477 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:55.232456    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fb926d1-1106-4438-bf30-3e6f37961b7c-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-97q4z\" (UID: \"7fb926d1-1106-4438-bf30-3e6f37961b7c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-97q4z"
Sep 17 21:54:55.232545 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:55.232524    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fb926d1-1106-4438-bf30-3e6f37961b7c-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-97q4z\" (UID: \"7fb926d1-1106-4438-bf30-3e6f37961b7c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-97q4z"
Sep 17 21:54:55.232646 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:55.232627    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fb926d1-1106-4438-bf30-3e6f37961b7c-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-97q4z\" (UID: \"7fb926d1-1106-4438-bf30-3e6f37961b7c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-97q4z"
Sep 17 21:54:55.234322 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:55.234305    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fb926d1-1106-4438-bf30-3e6f37961b7c-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-97q4z\" (UID: \"7fb926d1-1106-4438-bf30-3e6f37961b7c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-97q4z"
Sep 17 21:54:55.234590 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:55.234572    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fb926d1-1106-4438-bf30-3e6f37961b7c-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-97q4z\" (UID: \"7fb926d1-1106-4438-bf30-3e6f37961b7c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-97q4z"
Sep 17 21:54:55.240649 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:55.240630    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9frpf\" (UniqueName: \"kubernetes.io/projected/7fb926d1-1106-4438-bf30-3e6f37961b7c-kube-api-access-9frpf\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-97q4z\" (UID: \"7fb926d1-1106-4438-bf30-3e6f37961b7c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-97q4z"
Sep 17 21:54:55.321809 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:55.321759    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-97q4z"
Sep 17 21:54:55.594732 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:55.594707    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-97q4z"]
Sep 17 21:54:55.597150 ip-10-0-139-36 kubenswrapper[2575]: W0917 21:54:55.597120    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fb926d1_1106_4438_bf30_3e6f37961b7c.slice/crio-d4b510841789cd15a094d9abe1900838e1524d864c295074d40e2059f3e84a83 WatchSource:0}: Error finding container d4b510841789cd15a094d9abe1900838e1524d864c295074d40e2059f3e84a83: Status 404 returned error can't find the container with id d4b510841789cd15a094d9abe1900838e1524d864c295074d40e2059f3e84a83
Sep 17 21:54:56.195606 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:56.195571    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-97q4z" event={"ID":"7fb926d1-1106-4438-bf30-3e6f37961b7c","Type":"ContainerStarted","Data":"645af4103804dce1d69c1ff234992c527e9ce434f560815087aa7c11790c94a1"}
Sep 17 21:54:56.196070 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:56.195654    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-97q4z" event={"ID":"7fb926d1-1106-4438-bf30-3e6f37961b7c","Type":"ContainerStarted","Data":"d4b510841789cd15a094d9abe1900838e1524d864c295074d40e2059f3e84a83"}
Sep 17 21:54:56.215187 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:56.215136    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-97q4z" podStartSLOduration=2.215121556 podStartE2EDuration="2.215121556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:54:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:54:56.212885395 +0000 UTC m=+3961.365329762" watchObservedRunningTime="2026-09-17 21:54:56.215121556 +0000 UTC m=+3961.367565931"
Sep 17 21:54:56.322130 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:54:56.322097    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-97q4z"
Sep 17 21:55:04.919604 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:04.919562    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h9p96"]
Sep 17 21:55:04.922493 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:04.922459    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:55:04.923796 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:04.923769    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-97q4z"]
Sep 17 21:55:04.931187 ip-10-0-139-36 kubenswrapper[2575]: E0917 21:55:04.931144    2575 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 21:55:04.931187 ip-10-0-139-36 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:55:04.931187 ip-10-0-139-36 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 17 21:55:04.931187 ip-10-0-139-36 kubenswrapper[2575]:  > containerID="24e51c518f46f187ca23f7abaa1611ed6d1227ac5a1c65bfc94401fb24343d8b" cmd=["/bin/sleep","15"]
Sep 17 21:55:04.931418 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:04.931213    2575 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h9p96" message="" err=<
Sep 17 21:55:04.931418 ip-10-0-139-36 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:55:04.931418 ip-10-0-139-36 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 17 21:55:04.931418 ip-10-0-139-36 kubenswrapper[2575]:  >
Sep 17 21:55:04.931418 ip-10-0-139-36 kubenswrapper[2575]: E0917 21:55:04.931231    2575 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 21:55:04.931418 ip-10-0-139-36 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:55:04.931418 ip-10-0-139-36 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 17 21:55:04.931418 ip-10-0-139-36 kubenswrapper[2575]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h9p96" podUID="0d7fc31e-6876-43a5-8d90-7269f9f79d56" containerName="main" containerID="cri-o://24e51c518f46f187ca23f7abaa1611ed6d1227ac5a1c65bfc94401fb24343d8b"
Sep 17 21:55:04.931418 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:04.931291    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h9p96" podUID="0d7fc31e-6876-43a5-8d90-7269f9f79d56" containerName="main" containerID="cri-o://24e51c518f46f187ca23f7abaa1611ed6d1227ac5a1c65bfc94401fb24343d8b" gracePeriod=60
Sep 17 21:55:04.934430 ip-10-0-139-36 kubenswrapper[2575]: E0917 21:55:04.934346    2575 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 21:55:04.934430 ip-10-0-139-36 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:55:04.934430 ip-10-0-139-36 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 17 21:55:04.934430 ip-10-0-139-36 kubenswrapper[2575]:  > containerID="645af4103804dce1d69c1ff234992c527e9ce434f560815087aa7c11790c94a1" cmd=["/bin/sleep","15"]
Sep 17 21:55:04.934700 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:04.934448    2575 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-97q4z" message="" err=<
Sep 17 21:55:04.934700 ip-10-0-139-36 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:55:04.934700 ip-10-0-139-36 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 17 21:55:04.934700 ip-10-0-139-36 kubenswrapper[2575]:  >
Sep 17 21:55:04.934700 ip-10-0-139-36 kubenswrapper[2575]: E0917 21:55:04.934464    2575 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 21:55:04.934700 ip-10-0-139-36 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:55:04.934700 ip-10-0-139-36 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 17 21:55:04.934700 ip-10-0-139-36 kubenswrapper[2575]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-97q4z" podUID="7fb926d1-1106-4438-bf30-3e6f37961b7c" containerName="main" containerID="cri-o://645af4103804dce1d69c1ff234992c527e9ce434f560815087aa7c11790c94a1"
Sep 17 21:55:04.934700 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:04.934514    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-97q4z" podUID="7fb926d1-1106-4438-bf30-3e6f37961b7c" containerName="main" containerID="cri-o://645af4103804dce1d69c1ff234992c527e9ce434f560815087aa7c11790c94a1" gracePeriod=60
Sep 17 21:55:04.990818 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:04.990780    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:55:05.078631 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:05.078594    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h9p96" podUID="0d7fc31e-6876-43a5-8d90-7269f9f79d56" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Sep 17 21:55:05.187485 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:05.187454    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-97q4z"
Sep 17 21:55:05.244823 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:05.244790    2575 generic.go:356] "Generic (PLEG): container finished" podID="0d7fc31e-6876-43a5-8d90-7269f9f79d56" containerID="24e51c518f46f187ca23f7abaa1611ed6d1227ac5a1c65bfc94401fb24343d8b" exitCode=0
Sep 17 21:55:05.245039 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:05.244896    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h9p96" event={"ID":"0d7fc31e-6876-43a5-8d90-7269f9f79d56","Type":"ContainerDied","Data":"24e51c518f46f187ca23f7abaa1611ed6d1227ac5a1c65bfc94401fb24343d8b"}
Sep 17 21:55:05.246263 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:05.246239    2575 generic.go:356] "Generic (PLEG): container finished" podID="7fb926d1-1106-4438-bf30-3e6f37961b7c" containerID="645af4103804dce1d69c1ff234992c527e9ce434f560815087aa7c11790c94a1" exitCode=0
Sep 17 21:55:05.246405 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:05.246281    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-97q4z" event={"ID":"7fb926d1-1106-4438-bf30-3e6f37961b7c","Type":"ContainerDied","Data":"645af4103804dce1d69c1ff234992c527e9ce434f560815087aa7c11790c94a1"}
Sep 17 21:55:05.246405 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:05.246296    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-97q4z"
Sep 17 21:55:05.246405 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:05.246324    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-97q4z" event={"ID":"7fb926d1-1106-4438-bf30-3e6f37961b7c","Type":"ContainerDied","Data":"d4b510841789cd15a094d9abe1900838e1524d864c295074d40e2059f3e84a83"}
Sep 17 21:55:05.246405 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:05.246355    2575 scope.go:117] "RemoveContainer" containerID="645af4103804dce1d69c1ff234992c527e9ce434f560815087aa7c11790c94a1"
Sep 17 21:55:05.255195 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:05.255179    2575 scope.go:117] "RemoveContainer" containerID="645af4103804dce1d69c1ff234992c527e9ce434f560815087aa7c11790c94a1"
Sep 17 21:55:05.255506 ip-10-0-139-36 kubenswrapper[2575]: E0917 21:55:05.255489    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"645af4103804dce1d69c1ff234992c527e9ce434f560815087aa7c11790c94a1\": container with ID starting with 645af4103804dce1d69c1ff234992c527e9ce434f560815087aa7c11790c94a1 not found: ID does not exist" containerID="645af4103804dce1d69c1ff234992c527e9ce434f560815087aa7c11790c94a1"
Sep 17 21:55:05.255553 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:05.255511    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"645af4103804dce1d69c1ff234992c527e9ce434f560815087aa7c11790c94a1"} err="failed to get container status \"645af4103804dce1d69c1ff234992c527e9ce434f560815087aa7c11790c94a1\": rpc error: code = NotFound desc = could not find container \"645af4103804dce1d69c1ff234992c527e9ce434f560815087aa7c11790c94a1\": container with ID starting with 645af4103804dce1d69c1ff234992c527e9ce434f560815087aa7c11790c94a1 not found: ID does not exist"
Sep 17 21:55:05.290536 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:05.290510    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h9p96"
Sep 17 21:55:05.324482 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:05.324457    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9frpf\" (UniqueName: \"kubernetes.io/projected/7fb926d1-1106-4438-bf30-3e6f37961b7c-kube-api-access-9frpf\") pod \"7fb926d1-1106-4438-bf30-3e6f37961b7c\" (UID: \"7fb926d1-1106-4438-bf30-3e6f37961b7c\") "
Sep 17 21:55:05.324636 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:05.324489    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fb926d1-1106-4438-bf30-3e6f37961b7c-model-cache\") pod \"7fb926d1-1106-4438-bf30-3e6f37961b7c\" (UID: \"7fb926d1-1106-4438-bf30-3e6f37961b7c\") "
Sep 17 21:55:05.324636 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:05.324524    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fb926d1-1106-4438-bf30-3e6f37961b7c-dshm\") pod \"7fb926d1-1106-4438-bf30-3e6f37961b7c\" (UID: \"7fb926d1-1106-4438-bf30-3e6f37961b7c\") "
Sep 17 21:55:05.324636 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:05.324578    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fb926d1-1106-4438-bf30-3e6f37961b7c-tmp-dir\") pod \"7fb926d1-1106-4438-bf30-3e6f37961b7c\" (UID: \"7fb926d1-1106-4438-bf30-3e6f37961b7c\") "
Sep 17 21:55:05.324636 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:05.324631    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fb926d1-1106-4438-bf30-3e6f37961b7c-home\") pod \"7fb926d1-1106-4438-bf30-3e6f37961b7c\" (UID: \"7fb926d1-1106-4438-bf30-3e6f37961b7c\") "
Sep 17 21:55:05.324881 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:05.324676    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fb926d1-1106-4438-bf30-3e6f37961b7c-tls-certs\") pod \"7fb926d1-1106-4438-bf30-3e6f37961b7c\" (UID: \"7fb926d1-1106-4438-bf30-3e6f37961b7c\") "
Sep 17 21:55:05.324881 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:05.324783    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fb926d1-1106-4438-bf30-3e6f37961b7c-model-cache" (OuterVolumeSpecName: "model-cache") pod "7fb926d1-1106-4438-bf30-3e6f37961b7c" (UID: "7fb926d1-1106-4438-bf30-3e6f37961b7c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:55:05.324993 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:05.324933    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fb926d1-1106-4438-bf30-3e6f37961b7c-home" (OuterVolumeSpecName: "home") pod "7fb926d1-1106-4438-bf30-3e6f37961b7c" (UID: "7fb926d1-1106-4438-bf30-3e6f37961b7c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:55:05.324993 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:05.324945    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fb926d1-1106-4438-bf30-3e6f37961b7c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7fb926d1-1106-4438-bf30-3e6f37961b7c" (UID: "7fb926d1-1106-4438-bf30-3e6f37961b7c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:55:05.324993 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:05.324951    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fb926d1-1106-4438-bf30-3e6f37961b7c-model-cache\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:55:05.327030 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:05.327004    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fb926d1-1106-4438-bf30-3e6f37961b7c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7fb926d1-1106-4438-bf30-3e6f37961b7c" (UID: "7fb926d1-1106-4438-bf30-3e6f37961b7c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:55:05.327169 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:05.327147    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fb926d1-1106-4438-bf30-3e6f37961b7c-dshm" (OuterVolumeSpecName: "dshm") pod "7fb926d1-1106-4438-bf30-3e6f37961b7c" (UID: "7fb926d1-1106-4438-bf30-3e6f37961b7c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:55:05.327256 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:05.327244    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fb926d1-1106-4438-bf30-3e6f37961b7c-kube-api-access-9frpf" (OuterVolumeSpecName: "kube-api-access-9frpf") pod "7fb926d1-1106-4438-bf30-3e6f37961b7c" (UID: "7fb926d1-1106-4438-bf30-3e6f37961b7c"). InnerVolumeSpecName "kube-api-access-9frpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:55:05.425620 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:05.425535    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8dsss\" (UniqueName: \"kubernetes.io/projected/0d7fc31e-6876-43a5-8d90-7269f9f79d56-kube-api-access-8dsss\") pod \"0d7fc31e-6876-43a5-8d90-7269f9f79d56\" (UID: \"0d7fc31e-6876-43a5-8d90-7269f9f79d56\") "
Sep 17 21:55:05.425620 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:05.425602    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d7fc31e-6876-43a5-8d90-7269f9f79d56-tmp-dir\") pod \"0d7fc31e-6876-43a5-8d90-7269f9f79d56\" (UID: \"0d7fc31e-6876-43a5-8d90-7269f9f79d56\") "
Sep 17 21:55:05.425814 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:05.425642    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d7fc31e-6876-43a5-8d90-7269f9f79d56-model-cache\") pod \"0d7fc31e-6876-43a5-8d90-7269f9f79d56\" (UID: \"0d7fc31e-6876-43a5-8d90-7269f9f79d56\") "
Sep 17 21:55:05.425814 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:05.425661    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d7fc31e-6876-43a5-8d90-7269f9f79d56-dshm\") pod \"0d7fc31e-6876-43a5-8d90-7269f9f79d56\" (UID: \"0d7fc31e-6876-43a5-8d90-7269f9f79d56\") "
Sep 17 21:55:05.425814 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:05.425697    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d7fc31e-6876-43a5-8d90-7269f9f79d56-home\") pod \"0d7fc31e-6876-43a5-8d90-7269f9f79d56\" (UID: \"0d7fc31e-6876-43a5-8d90-7269f9f79d56\") "
Sep 17 21:55:05.425814 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:05.425724    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d7fc31e-6876-43a5-8d90-7269f9f79d56-tls-certs\") pod \"0d7fc31e-6876-43a5-8d90-7269f9f79d56\" (UID: \"0d7fc31e-6876-43a5-8d90-7269f9f79d56\") "
Sep 17 21:55:05.426080 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:05.425917    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d7fc31e-6876-43a5-8d90-7269f9f79d56-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0d7fc31e-6876-43a5-8d90-7269f9f79d56" (UID: "0d7fc31e-6876-43a5-8d90-7269f9f79d56"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:55:05.426080 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:05.425937    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d7fc31e-6876-43a5-8d90-7269f9f79d56-model-cache" (OuterVolumeSpecName: "model-cache") pod "0d7fc31e-6876-43a5-8d90-7269f9f79d56" (UID: "0d7fc31e-6876-43a5-8d90-7269f9f79d56"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:55:05.426080 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:05.426029    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d7fc31e-6876-43a5-8d90-7269f9f79d56-home" (OuterVolumeSpecName: "home") pod "0d7fc31e-6876-43a5-8d90-7269f9f79d56" (UID: "0d7fc31e-6876-43a5-8d90-7269f9f79d56"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:55:05.426244 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:05.426134    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fb926d1-1106-4438-bf30-3e6f37961b7c-home\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:55:05.426244 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:05.426151    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fb926d1-1106-4438-bf30-3e6f37961b7c-tls-certs\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:55:05.426244 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:05.426169    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9frpf\" (UniqueName: \"kubernetes.io/projected/7fb926d1-1106-4438-bf30-3e6f37961b7c-kube-api-access-9frpf\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:55:05.426244 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:05.426182    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d7fc31e-6876-43a5-8d90-7269f9f79d56-tmp-dir\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:55:05.426244 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:05.426195    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fb926d1-1106-4438-bf30-3e6f37961b7c-dshm\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:55:05.426244 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:05.426207    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d7fc31e-6876-43a5-8d90-7269f9f79d56-model-cache\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:55:05.426244 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:05.426220    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d7fc31e-6876-43a5-8d90-7269f9f79d56-home\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:55:05.426244 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:05.426233    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fb926d1-1106-4438-bf30-3e6f37961b7c-tmp-dir\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:55:05.427799 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:05.427778    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d7fc31e-6876-43a5-8d90-7269f9f79d56-dshm" (OuterVolumeSpecName: "dshm") pod "0d7fc31e-6876-43a5-8d90-7269f9f79d56" (UID: "0d7fc31e-6876-43a5-8d90-7269f9f79d56"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:55:05.427888 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:05.427833    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d7fc31e-6876-43a5-8d90-7269f9f79d56-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0d7fc31e-6876-43a5-8d90-7269f9f79d56" (UID: "0d7fc31e-6876-43a5-8d90-7269f9f79d56"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:55:05.427888 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:05.427838    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d7fc31e-6876-43a5-8d90-7269f9f79d56-kube-api-access-8dsss" (OuterVolumeSpecName: "kube-api-access-8dsss") pod "0d7fc31e-6876-43a5-8d90-7269f9f79d56" (UID: "0d7fc31e-6876-43a5-8d90-7269f9f79d56"). InnerVolumeSpecName "kube-api-access-8dsss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:55:05.527050 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:05.527015    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d7fc31e-6876-43a5-8d90-7269f9f79d56-dshm\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:55:05.527050 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:05.527043    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d7fc31e-6876-43a5-8d90-7269f9f79d56-tls-certs\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:55:05.527050 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:05.527061    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dsss\" (UniqueName: \"kubernetes.io/projected/0d7fc31e-6876-43a5-8d90-7269f9f79d56-kube-api-access-8dsss\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:55:05.563986 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:05.563948    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-97q4z"]
Sep 17 21:55:05.569228 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:05.569201    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-97q4z"]
Sep 17 21:55:06.253500 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:06.253415    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h9p96"
Sep 17 21:55:06.253500 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:06.253429    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h9p96" event={"ID":"0d7fc31e-6876-43a5-8d90-7269f9f79d56","Type":"ContainerDied","Data":"a8e071a3aabc49f08ea58f1317e1755abc9bc0bc60b406aa582dd0c7c15e7de8"}
Sep 17 21:55:06.253500 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:06.253484    2575 scope.go:117] "RemoveContainer" containerID="24e51c518f46f187ca23f7abaa1611ed6d1227ac5a1c65bfc94401fb24343d8b"
Sep 17 21:55:06.276347 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:06.276319    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h9p96"]
Sep 17 21:55:06.290034 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:06.284566    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h9p96"]
Sep 17 21:55:07.357550 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:07.357512    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d7fc31e-6876-43a5-8d90-7269f9f79d56" path="/var/lib/kubelet/pods/0d7fc31e-6876-43a5-8d90-7269f9f79d56/volumes"
Sep 17 21:55:07.357943 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:07.357929    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fb926d1-1106-4438-bf30-3e6f37961b7c" path="/var/lib/kubelet/pods/7fb926d1-1106-4438-bf30-3e6f37961b7c/volumes"
Sep 17 21:55:39.752164 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:39.752130    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-tk2t8"]
Sep 17 21:55:39.752700 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:39.752481    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d7fc31e-6876-43a5-8d90-7269f9f79d56" containerName="main"
Sep 17 21:55:39.752700 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:39.752493    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d7fc31e-6876-43a5-8d90-7269f9f79d56" containerName="main"
Sep 17 21:55:39.752700 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:39.752520    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fb926d1-1106-4438-bf30-3e6f37961b7c" containerName="main"
Sep 17 21:55:39.752700 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:39.752525    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fb926d1-1106-4438-bf30-3e6f37961b7c" containerName="main"
Sep 17 21:55:39.752700 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:39.752619    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d7fc31e-6876-43a5-8d90-7269f9f79d56" containerName="main"
Sep 17 21:55:39.752700 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:39.752637    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fb926d1-1106-4438-bf30-3e6f37961b7c" containerName="main"
Sep 17 21:55:39.755814 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:39.755796    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-tk2t8"
Sep 17 21:55:39.758461 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:39.758442    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 17 21:55:39.758597 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:39.758471    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 17 21:55:39.758597 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:39.758471    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 17 21:55:39.758597 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:39.758531    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 21:55:39.758597 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:39.758560    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-w6qcl\""
Sep 17 21:55:39.758832 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:39.758743    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Sep 17 21:55:39.772172 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:39.772150    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-tk2t8"]
Sep 17 21:55:39.835873 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:39.835828    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a68a5197-4398-42e4-9f58-63db8e11ef91-cabundle-cert\") pod \"prestop-hook-test-kserve-764979c8fb-tk2t8\" (UID: \"a68a5197-4398-42e4-9f58-63db8e11ef91\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-tk2t8"
Sep 17 21:55:39.836048 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:39.835898    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a68a5197-4398-42e4-9f58-63db8e11ef91-tmp-dir\") pod \"prestop-hook-test-kserve-764979c8fb-tk2t8\" (UID: \"a68a5197-4398-42e4-9f58-63db8e11ef91\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-tk2t8"
Sep 17 21:55:39.836048 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:39.835933    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a68a5197-4398-42e4-9f58-63db8e11ef91-kserve-provision-location\") pod \"prestop-hook-test-kserve-764979c8fb-tk2t8\" (UID: \"a68a5197-4398-42e4-9f58-63db8e11ef91\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-tk2t8"
Sep 17 21:55:39.836048 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:39.835959    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmdt5\" (UniqueName: \"kubernetes.io/projected/a68a5197-4398-42e4-9f58-63db8e11ef91-kube-api-access-dmdt5\") pod \"prestop-hook-test-kserve-764979c8fb-tk2t8\" (UID: \"a68a5197-4398-42e4-9f58-63db8e11ef91\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-tk2t8"
Sep 17 21:55:39.836048 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:39.835990    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a68a5197-4398-42e4-9f58-63db8e11ef91-home\") pod \"prestop-hook-test-kserve-764979c8fb-tk2t8\" (UID: \"a68a5197-4398-42e4-9f58-63db8e11ef91\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-tk2t8"
Sep 17 21:55:39.836048 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:39.836015    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a68a5197-4398-42e4-9f58-63db8e11ef91-tls-certs\") pod \"prestop-hook-test-kserve-764979c8fb-tk2t8\" (UID: \"a68a5197-4398-42e4-9f58-63db8e11ef91\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-tk2t8"
Sep 17 21:55:39.836222 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:39.836100    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a68a5197-4398-42e4-9f58-63db8e11ef91-model-cache\") pod \"prestop-hook-test-kserve-764979c8fb-tk2t8\" (UID: \"a68a5197-4398-42e4-9f58-63db8e11ef91\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-tk2t8"
Sep 17 21:55:39.836222 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:39.836140    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a68a5197-4398-42e4-9f58-63db8e11ef91-dshm\") pod \"prestop-hook-test-kserve-764979c8fb-tk2t8\" (UID: \"a68a5197-4398-42e4-9f58-63db8e11ef91\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-tk2t8"
Sep 17 21:55:39.936793 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:39.936750    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a68a5197-4398-42e4-9f58-63db8e11ef91-dshm\") pod \"prestop-hook-test-kserve-764979c8fb-tk2t8\" (UID: \"a68a5197-4398-42e4-9f58-63db8e11ef91\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-tk2t8"
Sep 17 21:55:39.937009 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:39.936802    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a68a5197-4398-42e4-9f58-63db8e11ef91-cabundle-cert\") pod \"prestop-hook-test-kserve-764979c8fb-tk2t8\" (UID: \"a68a5197-4398-42e4-9f58-63db8e11ef91\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-tk2t8"
Sep 17 21:55:39.937009 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:39.936839    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a68a5197-4398-42e4-9f58-63db8e11ef91-tmp-dir\") pod \"prestop-hook-test-kserve-764979c8fb-tk2t8\" (UID: \"a68a5197-4398-42e4-9f58-63db8e11ef91\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-tk2t8"
Sep 17 21:55:39.937009 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:39.936881    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a68a5197-4398-42e4-9f58-63db8e11ef91-kserve-provision-location\") pod \"prestop-hook-test-kserve-764979c8fb-tk2t8\" (UID: \"a68a5197-4398-42e4-9f58-63db8e11ef91\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-tk2t8"
Sep 17 21:55:39.937009 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:39.936914    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmdt5\" (UniqueName: \"kubernetes.io/projected/a68a5197-4398-42e4-9f58-63db8e11ef91-kube-api-access-dmdt5\") pod \"prestop-hook-test-kserve-764979c8fb-tk2t8\" (UID: \"a68a5197-4398-42e4-9f58-63db8e11ef91\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-tk2t8"
Sep 17 21:55:39.937262 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:39.937053    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a68a5197-4398-42e4-9f58-63db8e11ef91-home\") pod \"prestop-hook-test-kserve-764979c8fb-tk2t8\" (UID: \"a68a5197-4398-42e4-9f58-63db8e11ef91\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-tk2t8"
Sep 17 21:55:39.937262 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:39.937152    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a68a5197-4398-42e4-9f58-63db8e11ef91-tls-certs\") pod \"prestop-hook-test-kserve-764979c8fb-tk2t8\" (UID: \"a68a5197-4398-42e4-9f58-63db8e11ef91\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-tk2t8"
Sep 17 21:55:39.937382 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:39.937313    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a68a5197-4398-42e4-9f58-63db8e11ef91-tmp-dir\") pod \"prestop-hook-test-kserve-764979c8fb-tk2t8\" (UID: \"a68a5197-4398-42e4-9f58-63db8e11ef91\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-tk2t8"
Sep 17 21:55:39.937382 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:39.937361    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a68a5197-4398-42e4-9f58-63db8e11ef91-kserve-provision-location\") pod \"prestop-hook-test-kserve-764979c8fb-tk2t8\" (UID: \"a68a5197-4398-42e4-9f58-63db8e11ef91\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-tk2t8"
Sep 17 21:55:39.937468 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:39.937418    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a68a5197-4398-42e4-9f58-63db8e11ef91-home\") pod \"prestop-hook-test-kserve-764979c8fb-tk2t8\" (UID: \"a68a5197-4398-42e4-9f58-63db8e11ef91\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-tk2t8"
Sep 17 21:55:39.937652 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:39.937623    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a68a5197-4398-42e4-9f58-63db8e11ef91-model-cache\") pod \"prestop-hook-test-kserve-764979c8fb-tk2t8\" (UID: \"a68a5197-4398-42e4-9f58-63db8e11ef91\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-tk2t8"
Sep 17 21:55:39.937793 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:39.937739    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a68a5197-4398-42e4-9f58-63db8e11ef91-cabundle-cert\") pod \"prestop-hook-test-kserve-764979c8fb-tk2t8\" (UID: \"a68a5197-4398-42e4-9f58-63db8e11ef91\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-tk2t8"
Sep 17 21:55:39.937901 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:39.937884    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a68a5197-4398-42e4-9f58-63db8e11ef91-model-cache\") pod \"prestop-hook-test-kserve-764979c8fb-tk2t8\" (UID: \"a68a5197-4398-42e4-9f58-63db8e11ef91\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-tk2t8"
Sep 17 21:55:39.939200 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:39.939176    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a68a5197-4398-42e4-9f58-63db8e11ef91-dshm\") pod \"prestop-hook-test-kserve-764979c8fb-tk2t8\" (UID: \"a68a5197-4398-42e4-9f58-63db8e11ef91\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-tk2t8"
Sep 17 21:55:39.939746 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:39.939724    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a68a5197-4398-42e4-9f58-63db8e11ef91-tls-certs\") pod \"prestop-hook-test-kserve-764979c8fb-tk2t8\" (UID: \"a68a5197-4398-42e4-9f58-63db8e11ef91\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-tk2t8"
Sep 17 21:55:39.946659 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:39.946628    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmdt5\" (UniqueName: \"kubernetes.io/projected/a68a5197-4398-42e4-9f58-63db8e11ef91-kube-api-access-dmdt5\") pod \"prestop-hook-test-kserve-764979c8fb-tk2t8\" (UID: \"a68a5197-4398-42e4-9f58-63db8e11ef91\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-tk2t8"
Sep 17 21:55:40.066754 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:40.066662    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-tk2t8"
Sep 17 21:55:40.345924 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:40.345896    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-tk2t8"]
Sep 17 21:55:40.348276 ip-10-0-139-36 kubenswrapper[2575]: W0917 21:55:40.348247    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda68a5197_4398_42e4_9f58_63db8e11ef91.slice/crio-2a4290f22db612df55c26b914fb17e4168bc869f80797040fbff7e49af4fd7d6 WatchSource:0}: Error finding container 2a4290f22db612df55c26b914fb17e4168bc869f80797040fbff7e49af4fd7d6: Status 404 returned error can't find the container with id 2a4290f22db612df55c26b914fb17e4168bc869f80797040fbff7e49af4fd7d6
Sep 17 21:55:40.422270 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:40.422232    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-tk2t8" event={"ID":"a68a5197-4398-42e4-9f58-63db8e11ef91","Type":"ContainerStarted","Data":"373cd786702974fb0e3e061e6b82e8e13473e77ff4a173c88f2f87f028893928"}
Sep 17 21:55:40.422468 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:40.422291    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-tk2t8" event={"ID":"a68a5197-4398-42e4-9f58-63db8e11ef91","Type":"ContainerStarted","Data":"2a4290f22db612df55c26b914fb17e4168bc869f80797040fbff7e49af4fd7d6"}
Sep 17 21:55:41.442545 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:41.442501    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:55:43.439335 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:43.439297    2575 generic.go:356] "Generic (PLEG): container finished" podID="a68a5197-4398-42e4-9f58-63db8e11ef91" containerID="373cd786702974fb0e3e061e6b82e8e13473e77ff4a173c88f2f87f028893928" exitCode=0
Sep 17 21:55:43.439696 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:43.439353    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-tk2t8" event={"ID":"a68a5197-4398-42e4-9f58-63db8e11ef91","Type":"ContainerDied","Data":"373cd786702974fb0e3e061e6b82e8e13473e77ff4a173c88f2f87f028893928"}
Sep 17 21:55:44.447496 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:44.447455    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-tk2t8" event={"ID":"a68a5197-4398-42e4-9f58-63db8e11ef91","Type":"ContainerStarted","Data":"3255c3cc4f0d0766ed1e6ea38d572000adeca14815a2a279602282f50e37bb7e"}
Sep 17 21:55:44.473207 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:44.473150    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-tk2t8" podStartSLOduration=3.3525123519999998 podStartE2EDuration="5.473131096s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.120618744s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:55:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:55:44.472141721 +0000 UTC m=+4009.624586098" watchObservedRunningTime="2026-09-17 21:55:44.473131096 +0000 UTC m=+4009.625575487"
Sep 17 21:55:45.067621 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:45.067579    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-tk2t8"
Sep 17 21:55:50.067137 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:50.067097    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-tk2t8"
Sep 17 21:55:50.068378 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:50.068351    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-tk2t8" podUID="a68a5197-4398-42e4-9f58-63db8e11ef91" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Sep 17 21:55:52.077421 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:52.077382    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:55:58.026706 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:58.026667    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:55:58.034069 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:58.034027    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-tk2t8"]
Sep 17 21:55:58.123580 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:58.123536    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:55:59.110807 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:55:59.110766    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:56:05.194517 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:56:05.194473    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:56:05.308325 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:56:05.308275    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:56:13.061362 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:56:13.061291    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-tk2t8" podUID="a68a5197-4398-42e4-9f58-63db8e11ef91" containerName="main" containerID="cri-o://3255c3cc4f0d0766ed1e6ea38d572000adeca14815a2a279602282f50e37bb7e" gracePeriod=45
Sep 17 21:56:28.129067 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:56:28.128980    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:56:58.716229 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:56:58.716205    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-764979c8fb-tk2t8_a68a5197-4398-42e4-9f58-63db8e11ef91/main/0.log"
Sep 17 21:56:58.718843 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:56:58.718825    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-tk2t8"
Sep 17 21:56:58.765456 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:56:58.765428    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a68a5197-4398-42e4-9f58-63db8e11ef91-tmp-dir\") pod \"a68a5197-4398-42e4-9f58-63db8e11ef91\" (UID: \"a68a5197-4398-42e4-9f58-63db8e11ef91\") "
Sep 17 21:56:58.765572 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:56:58.765464    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a68a5197-4398-42e4-9f58-63db8e11ef91-tls-certs\") pod \"a68a5197-4398-42e4-9f58-63db8e11ef91\" (UID: \"a68a5197-4398-42e4-9f58-63db8e11ef91\") "
Sep 17 21:56:58.765572 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:56:58.765508    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dmdt5\" (UniqueName: \"kubernetes.io/projected/a68a5197-4398-42e4-9f58-63db8e11ef91-kube-api-access-dmdt5\") pod \"a68a5197-4398-42e4-9f58-63db8e11ef91\" (UID: \"a68a5197-4398-42e4-9f58-63db8e11ef91\") "
Sep 17 21:56:58.765572 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:56:58.765528    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a68a5197-4398-42e4-9f58-63db8e11ef91-home\") pod \"a68a5197-4398-42e4-9f58-63db8e11ef91\" (UID: \"a68a5197-4398-42e4-9f58-63db8e11ef91\") "
Sep 17 21:56:58.765572 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:56:58.765557    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a68a5197-4398-42e4-9f58-63db8e11ef91-model-cache\") pod \"a68a5197-4398-42e4-9f58-63db8e11ef91\" (UID: \"a68a5197-4398-42e4-9f58-63db8e11ef91\") "
Sep 17 21:56:58.765755 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:56:58.765635    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a68a5197-4398-42e4-9f58-63db8e11ef91-cabundle-cert\") pod \"a68a5197-4398-42e4-9f58-63db8e11ef91\" (UID: \"a68a5197-4398-42e4-9f58-63db8e11ef91\") "
Sep 17 21:56:58.765755 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:56:58.765684    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a68a5197-4398-42e4-9f58-63db8e11ef91-kserve-provision-location\") pod \"a68a5197-4398-42e4-9f58-63db8e11ef91\" (UID: \"a68a5197-4398-42e4-9f58-63db8e11ef91\") "
Sep 17 21:56:58.765755 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:56:58.765716    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a68a5197-4398-42e4-9f58-63db8e11ef91-dshm\") pod \"a68a5197-4398-42e4-9f58-63db8e11ef91\" (UID: \"a68a5197-4398-42e4-9f58-63db8e11ef91\") "
Sep 17 21:56:58.765940 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:56:58.765915    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a68a5197-4398-42e4-9f58-63db8e11ef91-model-cache" (OuterVolumeSpecName: "model-cache") pod "a68a5197-4398-42e4-9f58-63db8e11ef91" (UID: "a68a5197-4398-42e4-9f58-63db8e11ef91"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:56:58.766159 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:56:58.766135    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a68a5197-4398-42e4-9f58-63db8e11ef91-model-cache\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:56:58.766159 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:56:58.766142    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a68a5197-4398-42e4-9f58-63db8e11ef91-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a68a5197-4398-42e4-9f58-63db8e11ef91" (UID: "a68a5197-4398-42e4-9f58-63db8e11ef91"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:56:58.766368 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:56:58.766344    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a68a5197-4398-42e4-9f58-63db8e11ef91-home" (OuterVolumeSpecName: "home") pod "a68a5197-4398-42e4-9f58-63db8e11ef91" (UID: "a68a5197-4398-42e4-9f58-63db8e11ef91"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:56:58.766569 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:56:58.766540    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a68a5197-4398-42e4-9f58-63db8e11ef91-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a68a5197-4398-42e4-9f58-63db8e11ef91" (UID: "a68a5197-4398-42e4-9f58-63db8e11ef91"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:56:58.768194 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:56:58.768171    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a68a5197-4398-42e4-9f58-63db8e11ef91-kube-api-access-dmdt5" (OuterVolumeSpecName: "kube-api-access-dmdt5") pod "a68a5197-4398-42e4-9f58-63db8e11ef91" (UID: "a68a5197-4398-42e4-9f58-63db8e11ef91"). InnerVolumeSpecName "kube-api-access-dmdt5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:56:58.768304 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:56:58.768226    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a68a5197-4398-42e4-9f58-63db8e11ef91-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a68a5197-4398-42e4-9f58-63db8e11ef91" (UID: "a68a5197-4398-42e4-9f58-63db8e11ef91"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:56:58.768304 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:56:58.768271    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a68a5197-4398-42e4-9f58-63db8e11ef91-dshm" (OuterVolumeSpecName: "dshm") pod "a68a5197-4398-42e4-9f58-63db8e11ef91" (UID: "a68a5197-4398-42e4-9f58-63db8e11ef91"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:56:58.790817 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:56:58.790794    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a68a5197-4398-42e4-9f58-63db8e11ef91-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a68a5197-4398-42e4-9f58-63db8e11ef91" (UID: "a68a5197-4398-42e4-9f58-63db8e11ef91"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:56:58.828974 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:56:58.828927    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-764979c8fb-tk2t8_a68a5197-4398-42e4-9f58-63db8e11ef91/main/0.log"
Sep 17 21:56:58.829342 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:56:58.829319    2575 generic.go:356] "Generic (PLEG): container finished" podID="a68a5197-4398-42e4-9f58-63db8e11ef91" containerID="3255c3cc4f0d0766ed1e6ea38d572000adeca14815a2a279602282f50e37bb7e" exitCode=137
Sep 17 21:56:58.829400 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:56:58.829365    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-tk2t8"
Sep 17 21:56:58.829438 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:56:58.829403    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-tk2t8" event={"ID":"a68a5197-4398-42e4-9f58-63db8e11ef91","Type":"ContainerDied","Data":"3255c3cc4f0d0766ed1e6ea38d572000adeca14815a2a279602282f50e37bb7e"}
Sep 17 21:56:58.829478 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:56:58.829460    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-tk2t8" event={"ID":"a68a5197-4398-42e4-9f58-63db8e11ef91","Type":"ContainerDied","Data":"2a4290f22db612df55c26b914fb17e4168bc869f80797040fbff7e49af4fd7d6"}
Sep 17 21:56:58.829512 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:56:58.829485    2575 scope.go:117] "RemoveContainer" containerID="3255c3cc4f0d0766ed1e6ea38d572000adeca14815a2a279602282f50e37bb7e"
Sep 17 21:56:58.838793 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:56:58.838771    2575 scope.go:117] "RemoveContainer" containerID="373cd786702974fb0e3e061e6b82e8e13473e77ff4a173c88f2f87f028893928"
Sep 17 21:56:58.846820 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:56:58.846805    2575 scope.go:117] "RemoveContainer" containerID="3255c3cc4f0d0766ed1e6ea38d572000adeca14815a2a279602282f50e37bb7e"
Sep 17 21:56:58.847166 ip-10-0-139-36 kubenswrapper[2575]: E0917 21:56:58.847145    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3255c3cc4f0d0766ed1e6ea38d572000adeca14815a2a279602282f50e37bb7e\": container with ID starting with 3255c3cc4f0d0766ed1e6ea38d572000adeca14815a2a279602282f50e37bb7e not found: ID does not exist" containerID="3255c3cc4f0d0766ed1e6ea38d572000adeca14815a2a279602282f50e37bb7e"
Sep 17 21:56:58.847221 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:56:58.847173    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3255c3cc4f0d0766ed1e6ea38d572000adeca14815a2a279602282f50e37bb7e"} err="failed to get container status \"3255c3cc4f0d0766ed1e6ea38d572000adeca14815a2a279602282f50e37bb7e\": rpc error: code = NotFound desc = could not find container \"3255c3cc4f0d0766ed1e6ea38d572000adeca14815a2a279602282f50e37bb7e\": container with ID starting with 3255c3cc4f0d0766ed1e6ea38d572000adeca14815a2a279602282f50e37bb7e not found: ID does not exist"
Sep 17 21:56:58.847221 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:56:58.847188    2575 scope.go:117] "RemoveContainer" containerID="373cd786702974fb0e3e061e6b82e8e13473e77ff4a173c88f2f87f028893928"
Sep 17 21:56:58.847436 ip-10-0-139-36 kubenswrapper[2575]: E0917 21:56:58.847418    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"373cd786702974fb0e3e061e6b82e8e13473e77ff4a173c88f2f87f028893928\": container with ID starting with 373cd786702974fb0e3e061e6b82e8e13473e77ff4a173c88f2f87f028893928 not found: ID does not exist" containerID="373cd786702974fb0e3e061e6b82e8e13473e77ff4a173c88f2f87f028893928"
Sep 17 21:56:58.847484 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:56:58.847442    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"373cd786702974fb0e3e061e6b82e8e13473e77ff4a173c88f2f87f028893928"} err="failed to get container status \"373cd786702974fb0e3e061e6b82e8e13473e77ff4a173c88f2f87f028893928\": rpc error: code = NotFound desc = could not find container \"373cd786702974fb0e3e061e6b82e8e13473e77ff4a173c88f2f87f028893928\": container with ID starting with 373cd786702974fb0e3e061e6b82e8e13473e77ff4a173c88f2f87f028893928 not found: ID does not exist"
Sep 17 21:56:58.853338 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:56:58.853317    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-tk2t8"]
Sep 17 21:56:58.859346 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:56:58.859324    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-tk2t8"]
Sep 17 21:56:58.867417 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:56:58.867399    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmdt5\" (UniqueName: \"kubernetes.io/projected/a68a5197-4398-42e4-9f58-63db8e11ef91-kube-api-access-dmdt5\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:56:58.867478 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:56:58.867420    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a68a5197-4398-42e4-9f58-63db8e11ef91-home\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:56:58.867478 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:56:58.867430    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a68a5197-4398-42e4-9f58-63db8e11ef91-cabundle-cert\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:56:58.867478 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:56:58.867441    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a68a5197-4398-42e4-9f58-63db8e11ef91-kserve-provision-location\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:56:58.867478 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:56:58.867449    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a68a5197-4398-42e4-9f58-63db8e11ef91-dshm\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:56:58.867478 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:56:58.867457    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a68a5197-4398-42e4-9f58-63db8e11ef91-tmp-dir\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:56:58.867478 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:56:58.867464    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a68a5197-4398-42e4-9f58-63db8e11ef91-tls-certs\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:56:59.356787 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:56:59.356753    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a68a5197-4398-42e4-9f58-63db8e11ef91" path="/var/lib/kubelet/pods/a68a5197-4398-42e4-9f58-63db8e11ef91/volumes"
Sep 17 21:57:03.103032 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:03.102991    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:57:04.028267 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:04.028228    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-967v7/must-gather-qpzk6"]
Sep 17 21:57:04.028660 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:04.028642    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a68a5197-4398-42e4-9f58-63db8e11ef91" containerName="storage-initializer"
Sep 17 21:57:04.028708 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:04.028662    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a68a5197-4398-42e4-9f58-63db8e11ef91" containerName="storage-initializer"
Sep 17 21:57:04.028708 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:04.028683    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a68a5197-4398-42e4-9f58-63db8e11ef91" containerName="main"
Sep 17 21:57:04.028708 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:04.028689    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a68a5197-4398-42e4-9f58-63db8e11ef91" containerName="main"
Sep 17 21:57:04.028806 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:04.028748    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a68a5197-4398-42e4-9f58-63db8e11ef91" containerName="main"
Sep 17 21:57:04.033284 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:04.033265    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-967v7/must-gather-qpzk6"
Sep 17 21:57:04.036921 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:04.036871    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-967v7\"/\"kube-root-ca.crt\""
Sep 17 21:57:04.037266 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:04.037247    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-967v7\"/\"openshift-service-ca.crt\""
Sep 17 21:57:04.037365 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:04.037330    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-967v7\"/\"default-dockercfg-rdng6\""
Sep 17 21:57:04.052414 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:04.052386    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-967v7/must-gather-qpzk6"]
Sep 17 21:57:04.062279 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:04.062246    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:57:04.111726 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:04.111691    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdczd\" (UniqueName: \"kubernetes.io/projected/aa6453bb-b244-47a2-936f-25cd93882f8c-kube-api-access-bdczd\") pod \"must-gather-qpzk6\" (UID: \"aa6453bb-b244-47a2-936f-25cd93882f8c\") " pod="openshift-must-gather-967v7/must-gather-qpzk6"
Sep 17 21:57:04.111726 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:04.111728    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/aa6453bb-b244-47a2-936f-25cd93882f8c-must-gather-output\") pod \"must-gather-qpzk6\" (UID: \"aa6453bb-b244-47a2-936f-25cd93882f8c\") " pod="openshift-must-gather-967v7/must-gather-qpzk6"
Sep 17 21:57:04.147277 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:04.147243    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:57:04.213000 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:04.212968    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdczd\" (UniqueName: \"kubernetes.io/projected/aa6453bb-b244-47a2-936f-25cd93882f8c-kube-api-access-bdczd\") pod \"must-gather-qpzk6\" (UID: \"aa6453bb-b244-47a2-936f-25cd93882f8c\") " pod="openshift-must-gather-967v7/must-gather-qpzk6"
Sep 17 21:57:04.213000 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:04.213006    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/aa6453bb-b244-47a2-936f-25cd93882f8c-must-gather-output\") pod \"must-gather-qpzk6\" (UID: \"aa6453bb-b244-47a2-936f-25cd93882f8c\") " pod="openshift-must-gather-967v7/must-gather-qpzk6"
Sep 17 21:57:04.213412 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:04.213391    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/aa6453bb-b244-47a2-936f-25cd93882f8c-must-gather-output\") pod \"must-gather-qpzk6\" (UID: \"aa6453bb-b244-47a2-936f-25cd93882f8c\") " pod="openshift-must-gather-967v7/must-gather-qpzk6"
Sep 17 21:57:04.221460 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:04.221436    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdczd\" (UniqueName: \"kubernetes.io/projected/aa6453bb-b244-47a2-936f-25cd93882f8c-kube-api-access-bdczd\") pod \"must-gather-qpzk6\" (UID: \"aa6453bb-b244-47a2-936f-25cd93882f8c\") " pod="openshift-must-gather-967v7/must-gather-qpzk6"
Sep 17 21:57:04.342443 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:04.342347    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-967v7/must-gather-qpzk6"
Sep 17 21:57:04.616010 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:04.615979    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-967v7/must-gather-qpzk6"]
Sep 17 21:57:04.616774 ip-10-0-139-36 kubenswrapper[2575]: W0917 21:57:04.616749    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaa6453bb_b244_47a2_936f_25cd93882f8c.slice/crio-505ba9d9da52d2f74b343add9b84b126d71249940442d220c2582d638988c90c WatchSource:0}: Error finding container 505ba9d9da52d2f74b343add9b84b126d71249940442d220c2582d638988c90c: Status 404 returned error can't find the container with id 505ba9d9da52d2f74b343add9b84b126d71249940442d220c2582d638988c90c
Sep 17 21:57:04.618509 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:04.618493    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 21:57:04.861605 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:04.861568    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-967v7/must-gather-qpzk6" event={"ID":"aa6453bb-b244-47a2-936f-25cd93882f8c","Type":"ContainerStarted","Data":"505ba9d9da52d2f74b343add9b84b126d71249940442d220c2582d638988c90c"}
Sep 17 21:57:09.349673 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:09.349635    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:57:11.906875 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:11.906822    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-967v7/must-gather-qpzk6" event={"ID":"aa6453bb-b244-47a2-936f-25cd93882f8c","Type":"ContainerStarted","Data":"1ace724c6be2e756cb191b8fb089a40b2064928e35c000b59dcb74085e620de6"}
Sep 17 21:57:11.907283 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:11.906891    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-967v7/must-gather-qpzk6" event={"ID":"aa6453bb-b244-47a2-936f-25cd93882f8c","Type":"ContainerStarted","Data":"6af3a01b01ed726599aa0cee8b01db89d7a98c2234512470af90187279f81342"}
Sep 17 21:57:11.923149 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:11.923090    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-967v7/must-gather-qpzk6" podStartSLOduration=1.657329563 podStartE2EDuration="7.923063501s" totalImagesPullingTime="6.265733938s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:57:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:57:11.922890201 +0000 UTC m=+4097.075334581" watchObservedRunningTime="2026-09-17 21:57:11.923063501 +0000 UTC m=+4097.075507879"
Sep 17 21:57:15.393029 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:15.392982    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:57:15.528494 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:15.528450    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:57:28.131242 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:28.131193    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-92k25_bc057dcf-d949-4624-b5c7-631b60bbc740/istio-proxy/0.log"
Sep 17 21:57:28.153972 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:28.153939    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-92k25_bc057dcf-d949-4624-b5c7-631b60bbc740/istio-proxy/1.log"
Sep 17 21:57:28.172459 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:28.172431    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8b75c8b68-8wx4k_7eb327ea-1cee-4481-8533-423762e782e0/router/0.log"
Sep 17 21:57:29.012363 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:29.012327    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-92k25_bc057dcf-d949-4624-b5c7-631b60bbc740/istio-proxy/0.log"
Sep 17 21:57:29.035791 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:29.035731    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-92k25_bc057dcf-d949-4624-b5c7-631b60bbc740/istio-proxy/1.log"
Sep 17 21:57:29.059321 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:29.059298    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8b75c8b68-8wx4k_7eb327ea-1cee-4481-8533-423762e782e0/router/0.log"
Sep 17 21:57:30.094343 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:30.094302    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-fw2sk_3863cd4c-03d9-4b97-ad6f-73b0cd07cb3f/authorino/0.log"
Sep 17 21:57:33.028589 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:33.028488    2575 generic.go:356] "Generic (PLEG): container finished" podID="aa6453bb-b244-47a2-936f-25cd93882f8c" containerID="6af3a01b01ed726599aa0cee8b01db89d7a98c2234512470af90187279f81342" exitCode=0
Sep 17 21:57:33.029051 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:33.028615    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-967v7/must-gather-qpzk6" event={"ID":"aa6453bb-b244-47a2-936f-25cd93882f8c","Type":"ContainerDied","Data":"6af3a01b01ed726599aa0cee8b01db89d7a98c2234512470af90187279f81342"}
Sep 17 21:57:33.029051 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:33.028978    2575 scope.go:117] "RemoveContainer" containerID="6af3a01b01ed726599aa0cee8b01db89d7a98c2234512470af90187279f81342"
Sep 17 21:57:33.729352 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:33.729323    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-967v7_must-gather-qpzk6_aa6453bb-b244-47a2-936f-25cd93882f8c/gather/0.log"
Sep 17 21:57:34.178390 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:34.178295    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:57:34.362487 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:34.362443    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:57:34.447696 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:34.447672    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:57:37.389227 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:37.389172    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-dpwhq_334a13ac-ccb2-43eb-bd43-566981c058ff/global-pull-secret-syncer/0.log"
Sep 17 21:57:37.506202 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:37.506173    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log"
Sep 17 21:57:37.674412 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:37.674290    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-36.ec2.internal_21ea1fa7387769fb888cd296b1eefe37/haproxy/0.log"
Sep 17 21:57:39.153129 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:39.153092    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-967v7/must-gather-qpzk6"]
Sep 17 21:57:39.153617 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:39.153267    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-967v7/must-gather-qpzk6" podUID="aa6453bb-b244-47a2-936f-25cd93882f8c" containerName="copy" containerID="cri-o://1ace724c6be2e756cb191b8fb089a40b2064928e35c000b59dcb74085e620de6" gracePeriod=2
Sep 17 21:57:39.160256 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:39.160218    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-967v7/must-gather-qpzk6"]
Sep 17 21:57:39.160778 ip-10-0-139-36 kubenswrapper[2575]: E0917 21:57:39.160743    2575 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"aa6453bb-b244-47a2-936f-25cd93882f8c\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"openshift-must-gather-967v7\"/\"must-gather-qpzk6\": pods \"must-gather-qpzk6\" not found" pod="openshift-must-gather-967v7/must-gather-qpzk6"
Sep 17 21:57:39.389770 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:39.389749    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-967v7_must-gather-qpzk6_aa6453bb-b244-47a2-936f-25cd93882f8c/copy/0.log"
Sep 17 21:57:39.392423 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:39.392406    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-967v7/must-gather-qpzk6"
Sep 17 21:57:39.440200 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:39.440131    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/aa6453bb-b244-47a2-936f-25cd93882f8c-must-gather-output\") pod \"aa6453bb-b244-47a2-936f-25cd93882f8c\" (UID: \"aa6453bb-b244-47a2-936f-25cd93882f8c\") "
Sep 17 21:57:39.440309 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:39.440216    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bdczd\" (UniqueName: \"kubernetes.io/projected/aa6453bb-b244-47a2-936f-25cd93882f8c-kube-api-access-bdczd\") pod \"aa6453bb-b244-47a2-936f-25cd93882f8c\" (UID: \"aa6453bb-b244-47a2-936f-25cd93882f8c\") "
Sep 17 21:57:39.442562 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:39.442532    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa6453bb-b244-47a2-936f-25cd93882f8c-kube-api-access-bdczd" (OuterVolumeSpecName: "kube-api-access-bdczd") pod "aa6453bb-b244-47a2-936f-25cd93882f8c" (UID: "aa6453bb-b244-47a2-936f-25cd93882f8c"). InnerVolumeSpecName "kube-api-access-bdczd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:57:39.445941 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:39.445915    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa6453bb-b244-47a2-936f-25cd93882f8c-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "aa6453bb-b244-47a2-936f-25cd93882f8c" (UID: "aa6453bb-b244-47a2-936f-25cd93882f8c"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:57:39.541686 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:39.541653    2575 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/aa6453bb-b244-47a2-936f-25cd93882f8c-must-gather-output\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:57:39.541686 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:39.541681    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdczd\" (UniqueName: \"kubernetes.io/projected/aa6453bb-b244-47a2-936f-25cd93882f8c-kube-api-access-bdczd\") on node \"ip-10-0-139-36.ec2.internal\" DevicePath \"\""
Sep 17 21:57:40.063300 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:40.063273    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-967v7_must-gather-qpzk6_aa6453bb-b244-47a2-936f-25cd93882f8c/copy/0.log"
Sep 17 21:57:40.063702 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:40.063678    2575 generic.go:356] "Generic (PLEG): container finished" podID="aa6453bb-b244-47a2-936f-25cd93882f8c" containerID="1ace724c6be2e756cb191b8fb089a40b2064928e35c000b59dcb74085e620de6" exitCode=143
Sep 17 21:57:40.063780 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:40.063721    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-967v7/must-gather-qpzk6"
Sep 17 21:57:40.063822 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:40.063781    2575 scope.go:117] "RemoveContainer" containerID="1ace724c6be2e756cb191b8fb089a40b2064928e35c000b59dcb74085e620de6"
Sep 17 21:57:40.072099 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:40.072081    2575 scope.go:117] "RemoveContainer" containerID="6af3a01b01ed726599aa0cee8b01db89d7a98c2234512470af90187279f81342"
Sep 17 21:57:40.084098 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:40.084080    2575 scope.go:117] "RemoveContainer" containerID="1ace724c6be2e756cb191b8fb089a40b2064928e35c000b59dcb74085e620de6"
Sep 17 21:57:40.084374 ip-10-0-139-36 kubenswrapper[2575]: E0917 21:57:40.084355    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ace724c6be2e756cb191b8fb089a40b2064928e35c000b59dcb74085e620de6\": container with ID starting with 1ace724c6be2e756cb191b8fb089a40b2064928e35c000b59dcb74085e620de6 not found: ID does not exist" containerID="1ace724c6be2e756cb191b8fb089a40b2064928e35c000b59dcb74085e620de6"
Sep 17 21:57:40.084433 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:40.084385    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ace724c6be2e756cb191b8fb089a40b2064928e35c000b59dcb74085e620de6"} err="failed to get container status \"1ace724c6be2e756cb191b8fb089a40b2064928e35c000b59dcb74085e620de6\": rpc error: code = NotFound desc = could not find container \"1ace724c6be2e756cb191b8fb089a40b2064928e35c000b59dcb74085e620de6\": container with ID starting with 1ace724c6be2e756cb191b8fb089a40b2064928e35c000b59dcb74085e620de6 not found: ID does not exist"
Sep 17 21:57:40.084433 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:40.084403    2575 scope.go:117] "RemoveContainer" containerID="6af3a01b01ed726599aa0cee8b01db89d7a98c2234512470af90187279f81342"
Sep 17 21:57:40.084631 ip-10-0-139-36 kubenswrapper[2575]: E0917 21:57:40.084618    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6af3a01b01ed726599aa0cee8b01db89d7a98c2234512470af90187279f81342\": container with ID starting with 6af3a01b01ed726599aa0cee8b01db89d7a98c2234512470af90187279f81342 not found: ID does not exist" containerID="6af3a01b01ed726599aa0cee8b01db89d7a98c2234512470af90187279f81342"
Sep 17 21:57:40.084677 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:40.084635    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6af3a01b01ed726599aa0cee8b01db89d7a98c2234512470af90187279f81342"} err="failed to get container status \"6af3a01b01ed726599aa0cee8b01db89d7a98c2234512470af90187279f81342\": rpc error: code = NotFound desc = could not find container \"6af3a01b01ed726599aa0cee8b01db89d7a98c2234512470af90187279f81342\": container with ID starting with 6af3a01b01ed726599aa0cee8b01db89d7a98c2234512470af90187279f81342 not found: ID does not exist"
Sep 17 21:57:40.941030 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:40.940991    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:57:41.357072 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:41.357043    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa6453bb-b244-47a2-936f-25cd93882f8c" path="/var/lib/kubelet/pods/aa6453bb-b244-47a2-936f-25cd93882f8c/volumes"
Sep 17 21:57:42.032144 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:42.032100    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mk66j_1a25b3d5-790d-41a8-bea7-4689843c7d71/konnectivity-agent/0.log" limit=1024
Sep 17 21:57:42.479416 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:42.479383    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-fw2sk_3863cd4c-03d9-4b97-ad6f-73b0cd07cb3f/authorino/0.log"
Sep 17 21:57:44.049104 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:44.049066    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5ab4b357-f52a-4aa0-8cb2-27e6c5758243/alertmanager/0.log"
Sep 17 21:57:44.076899 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:44.076870    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5ab4b357-f52a-4aa0-8cb2-27e6c5758243/config-reloader/0.log"
Sep 17 21:57:44.099173 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:44.099146    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5ab4b357-f52a-4aa0-8cb2-27e6c5758243/kube-rbac-proxy-web/0.log"
Sep 17 21:57:44.124443 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:44.124420    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5ab4b357-f52a-4aa0-8cb2-27e6c5758243/kube-rbac-proxy/0.log"
Sep 17 21:57:44.146826 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:44.146808    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5ab4b357-f52a-4aa0-8cb2-27e6c5758243/kube-rbac-proxy-metric/0.log"
Sep 17 21:57:44.167196 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:44.167175    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5ab4b357-f52a-4aa0-8cb2-27e6c5758243/prom-label-proxy/0.log"
Sep 17 21:57:44.186559 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:44.186538    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5ab4b357-f52a-4aa0-8cb2-27e6c5758243/init-config-reloader/0.log"
Sep 17 21:57:44.251743 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:44.251723    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-847b8bd8d7-lm54d_a95f9e39-2b33-434d-b6d0-807da6f8f4d2/cluster-monitoring-operator/0.log"
Sep 17 21:57:44.291122 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:44.291091    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-9gdxc_0205bdac-de0b-423b-a6d5-e515f40473f1/kube-state-metrics/0.log"
Sep 17 21:57:44.311841 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:44.311772    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-9gdxc_0205bdac-de0b-423b-a6d5-e515f40473f1/kube-rbac-proxy-main/0.log"
Sep 17 21:57:44.342998 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:44.342980    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-9gdxc_0205bdac-de0b-423b-a6d5-e515f40473f1/kube-rbac-proxy-self/0.log"
Sep 17 21:57:44.373404 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:44.373386    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-67bbf8d54-844pn_54f14609-0f14-4c0f-87f7-aebeba7a2b0e/metrics-server/0.log"
Sep 17 21:57:44.401943 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:44.401923    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5984459ff8-nrqnb_9dbbca3f-dbb6-40b3-9553-b710eddde112/monitoring-plugin/0.log"
Sep 17 21:57:44.508889 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:44.508866    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kb9r5_80762970-c96e-4ab0-b6e8-c9c7107ed1fc/node-exporter/0.log"
Sep 17 21:57:44.529241 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:44.529218    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kb9r5_80762970-c96e-4ab0-b6e8-c9c7107ed1fc/kube-rbac-proxy/0.log"
Sep 17 21:57:44.548113 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:44.548097    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kb9r5_80762970-c96e-4ab0-b6e8-c9c7107ed1fc/init-textfile/0.log"
Sep 17 21:57:44.993819 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:44.993740    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-pp9dx_ecd1e7d2-5f90-44ea-a0d4-a8a3c14e90d5/prometheus-operator/0.log"
Sep 17 21:57:45.018669 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:45.012678    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-pp9dx_ecd1e7d2-5f90-44ea-a0d4-a8a3c14e90d5/kube-rbac-proxy/0.log"
Sep 17 21:57:45.074476 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:45.074447    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-685dfdf6c8-5n6gw_b4685e11-32bc-4200-8737-4dfecd8f7698/telemeter-client/0.log"
Sep 17 21:57:45.113110 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:45.113083    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-685dfdf6c8-5n6gw_b4685e11-32bc-4200-8737-4dfecd8f7698/reload/0.log"
Sep 17 21:57:45.133550 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:45.133525    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-685dfdf6c8-5n6gw_b4685e11-32bc-4200-8737-4dfecd8f7698/kube-rbac-proxy/0.log"
Sep 17 21:57:46.545253 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:46.545217    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6446c8d75d-gjz76_f88bd94f-31d4-4c7a-a22d-0be24fae956e/networking-console-plugin/0.log"
Sep 17 21:57:47.244781 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:47.244749    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-j8tzf/perf-node-gather-daemonset-csh9j"]
Sep 17 21:57:47.245114 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:47.245100    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa6453bb-b244-47a2-936f-25cd93882f8c" containerName="gather"
Sep 17 21:57:47.245165 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:47.245116    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa6453bb-b244-47a2-936f-25cd93882f8c" containerName="gather"
Sep 17 21:57:47.245165 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:47.245138    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa6453bb-b244-47a2-936f-25cd93882f8c" containerName="copy"
Sep 17 21:57:47.245165 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:47.245143    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa6453bb-b244-47a2-936f-25cd93882f8c" containerName="copy"
Sep 17 21:57:47.245260 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:47.245218    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa6453bb-b244-47a2-936f-25cd93882f8c" containerName="gather"
Sep 17 21:57:47.245260 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:47.245227    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa6453bb-b244-47a2-936f-25cd93882f8c" containerName="copy"
Sep 17 21:57:47.248349 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:47.248333    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j8tzf/perf-node-gather-daemonset-csh9j"
Sep 17 21:57:47.255157 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:47.255123    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-j8tzf\"/\"openshift-service-ca.crt\""
Sep 17 21:57:47.255157 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:47.255139    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-j8tzf\"/\"default-dockercfg-4wxnw\""
Sep 17 21:57:47.255915 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:47.255895    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-j8tzf\"/\"kube-root-ca.crt\""
Sep 17 21:57:47.264228 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:47.264205    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j8tzf/perf-node-gather-daemonset-csh9j"]
Sep 17 21:57:47.301556 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:47.301527    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9cbb9c86-39a7-48dc-8f1a-72e929eb5796-proc\") pod \"perf-node-gather-daemonset-csh9j\" (UID: \"9cbb9c86-39a7-48dc-8f1a-72e929eb5796\") " pod="openshift-must-gather-j8tzf/perf-node-gather-daemonset-csh9j"
Sep 17 21:57:47.301666 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:47.301560    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9cbb9c86-39a7-48dc-8f1a-72e929eb5796-podres\") pod \"perf-node-gather-daemonset-csh9j\" (UID: \"9cbb9c86-39a7-48dc-8f1a-72e929eb5796\") " pod="openshift-must-gather-j8tzf/perf-node-gather-daemonset-csh9j"
Sep 17 21:57:47.301666 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:47.301589    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9cbb9c86-39a7-48dc-8f1a-72e929eb5796-sys\") pod \"perf-node-gather-daemonset-csh9j\" (UID: \"9cbb9c86-39a7-48dc-8f1a-72e929eb5796\") " pod="openshift-must-gather-j8tzf/perf-node-gather-daemonset-csh9j"
Sep 17 21:57:47.301666 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:47.301648    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9cbb9c86-39a7-48dc-8f1a-72e929eb5796-lib-modules\") pod \"perf-node-gather-daemonset-csh9j\" (UID: \"9cbb9c86-39a7-48dc-8f1a-72e929eb5796\") " pod="openshift-must-gather-j8tzf/perf-node-gather-daemonset-csh9j"
Sep 17 21:57:47.301813 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:47.301717    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrtcb\" (UniqueName: \"kubernetes.io/projected/9cbb9c86-39a7-48dc-8f1a-72e929eb5796-kube-api-access-hrtcb\") pod \"perf-node-gather-daemonset-csh9j\" (UID: \"9cbb9c86-39a7-48dc-8f1a-72e929eb5796\") " pod="openshift-must-gather-j8tzf/perf-node-gather-daemonset-csh9j"
Sep 17 21:57:47.403041 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:47.403018    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9cbb9c86-39a7-48dc-8f1a-72e929eb5796-lib-modules\") pod \"perf-node-gather-daemonset-csh9j\" (UID: \"9cbb9c86-39a7-48dc-8f1a-72e929eb5796\") " pod="openshift-must-gather-j8tzf/perf-node-gather-daemonset-csh9j"
Sep 17 21:57:47.403203 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:47.403185    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrtcb\" (UniqueName: \"kubernetes.io/projected/9cbb9c86-39a7-48dc-8f1a-72e929eb5796-kube-api-access-hrtcb\") pod \"perf-node-gather-daemonset-csh9j\" (UID: \"9cbb9c86-39a7-48dc-8f1a-72e929eb5796\") " pod="openshift-must-gather-j8tzf/perf-node-gather-daemonset-csh9j"
Sep 17 21:57:47.403259 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:47.403202    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9cbb9c86-39a7-48dc-8f1a-72e929eb5796-lib-modules\") pod \"perf-node-gather-daemonset-csh9j\" (UID: \"9cbb9c86-39a7-48dc-8f1a-72e929eb5796\") " pod="openshift-must-gather-j8tzf/perf-node-gather-daemonset-csh9j"
Sep 17 21:57:47.403259 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:47.403227    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9cbb9c86-39a7-48dc-8f1a-72e929eb5796-proc\") pod \"perf-node-gather-daemonset-csh9j\" (UID: \"9cbb9c86-39a7-48dc-8f1a-72e929eb5796\") " pod="openshift-must-gather-j8tzf/perf-node-gather-daemonset-csh9j"
Sep 17 21:57:47.403259 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:47.403244    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9cbb9c86-39a7-48dc-8f1a-72e929eb5796-podres\") pod \"perf-node-gather-daemonset-csh9j\" (UID: \"9cbb9c86-39a7-48dc-8f1a-72e929eb5796\") " pod="openshift-must-gather-j8tzf/perf-node-gather-daemonset-csh9j"
Sep 17 21:57:47.403382 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:47.403303    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9cbb9c86-39a7-48dc-8f1a-72e929eb5796-proc\") pod \"perf-node-gather-daemonset-csh9j\" (UID: \"9cbb9c86-39a7-48dc-8f1a-72e929eb5796\") " pod="openshift-must-gather-j8tzf/perf-node-gather-daemonset-csh9j"
Sep 17 21:57:47.403382 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:47.403330    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9cbb9c86-39a7-48dc-8f1a-72e929eb5796-podres\") pod \"perf-node-gather-daemonset-csh9j\" (UID: \"9cbb9c86-39a7-48dc-8f1a-72e929eb5796\") " pod="openshift-must-gather-j8tzf/perf-node-gather-daemonset-csh9j"
Sep 17 21:57:47.403382 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:47.403338    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9cbb9c86-39a7-48dc-8f1a-72e929eb5796-sys\") pod \"perf-node-gather-daemonset-csh9j\" (UID: \"9cbb9c86-39a7-48dc-8f1a-72e929eb5796\") " pod="openshift-must-gather-j8tzf/perf-node-gather-daemonset-csh9j"
Sep 17 21:57:47.403382 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:47.403374    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9cbb9c86-39a7-48dc-8f1a-72e929eb5796-sys\") pod \"perf-node-gather-daemonset-csh9j\" (UID: \"9cbb9c86-39a7-48dc-8f1a-72e929eb5796\") " pod="openshift-must-gather-j8tzf/perf-node-gather-daemonset-csh9j"
Sep 17 21:57:47.412462 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:47.412440    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrtcb\" (UniqueName: \"kubernetes.io/projected/9cbb9c86-39a7-48dc-8f1a-72e929eb5796-kube-api-access-hrtcb\") pod \"perf-node-gather-daemonset-csh9j\" (UID: \"9cbb9c86-39a7-48dc-8f1a-72e929eb5796\") " pod="openshift-must-gather-j8tzf/perf-node-gather-daemonset-csh9j"
Sep 17 21:57:47.558177 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:47.558104    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j8tzf/perf-node-gather-daemonset-csh9j"
Sep 17 21:57:47.828016 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:47.827993    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j8tzf/perf-node-gather-daemonset-csh9j"]
Sep 17 21:57:47.830059 ip-10-0-139-36 kubenswrapper[2575]: W0917 21:57:47.830026    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9cbb9c86_39a7_48dc_8f1a_72e929eb5796.slice/crio-14afc1ea54c4b4e223763607796c9b3bf9eab761227b4a356865ae143a132e54 WatchSource:0}: Error finding container 14afc1ea54c4b4e223763607796c9b3bf9eab761227b4a356865ae143a132e54: Status 404 returned error can't find the container with id 14afc1ea54c4b4e223763607796c9b3bf9eab761227b4a356865ae143a132e54
Sep 17 21:57:48.104830 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:48.104734    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j8tzf/perf-node-gather-daemonset-csh9j" event={"ID":"9cbb9c86-39a7-48dc-8f1a-72e929eb5796","Type":"ContainerStarted","Data":"b452774cd582aaee2b4a930c53aded14c552429dcc20353178ac973674961b45"}
Sep 17 21:57:48.104830 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:48.104795    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j8tzf/perf-node-gather-daemonset-csh9j" event={"ID":"9cbb9c86-39a7-48dc-8f1a-72e929eb5796","Type":"ContainerStarted","Data":"14afc1ea54c4b4e223763607796c9b3bf9eab761227b4a356865ae143a132e54"}
Sep 17 21:57:48.105023 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:48.104952    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-j8tzf/perf-node-gather-daemonset-csh9j"
Sep 17 21:57:48.124554 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:48.124508    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-j8tzf/perf-node-gather-daemonset-csh9j" podStartSLOduration=1.124494737 podStartE2EDuration="1.124494737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:57:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:57:48.122378321 +0000 UTC m=+4133.274822697" watchObservedRunningTime="2026-09-17 21:57:48.124494737 +0000 UTC m=+4133.276939110"
Sep 17 21:57:49.290417 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:49.290389    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-d7m9q_3afbe98f-5052-44e2-9d4b-da4bd2d30c3a/dns/0.log"
Sep 17 21:57:49.309504 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:49.309477    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-d7m9q_3afbe98f-5052-44e2-9d4b-da4bd2d30c3a/kube-rbac-proxy/0.log"
Sep 17 21:57:49.329953 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:49.329934    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-48p9f_5af35da6-3bf6-49ca-9a78-0218c7166c83/dns-node-resolver/0.log"
Sep 17 21:57:49.947225 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:49.947193    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2lrgx_0369efa0-b31a-4fa5-b426-95ba067ea731/node-ca/0.log"
Sep 17 21:57:51.120916 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:51.120888    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-92k25_bc057dcf-d949-4624-b5c7-631b60bbc740/istio-proxy/0.log"
Sep 17 21:57:51.148643 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:51.148613    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-92k25_bc057dcf-d949-4624-b5c7-631b60bbc740/istio-proxy/1.log"
Sep 17 21:57:51.210325 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:51.210297    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8b75c8b68-8wx4k_7eb327ea-1cee-4481-8533-423762e782e0/router/0.log"
Sep 17 21:57:51.789155 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:51.789122    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-z4lgm_54ba4bcf-1779-4a9d-a5c3-d3bc4507466c/serve-healthcheck-canary/0.log"
Sep 17 21:57:52.334677 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:52.334636    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-jknl5_7e30e07a-26a2-4051-9dba-6170acb64eba/insights-operator/0.log"
Sep 17 21:57:52.337592 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:52.337572    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-jknl5_7e30e07a-26a2-4051-9dba-6170acb64eba/insights-operator/1.log"
Sep 17 21:57:52.510462 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:52.510436    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qsltg_fc616903-063e-4c1c-b98e-472c4be8203f/kube-rbac-proxy/0.log"
Sep 17 21:57:52.532140 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:52.532114    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qsltg_fc616903-063e-4c1c-b98e-472c4be8203f/exporter/0.log"
Sep 17 21:57:52.559950 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:52.559900    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qsltg_fc616903-063e-4c1c-b98e-472c4be8203f/extractor/0.log"
Sep 17 21:57:54.119501 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:54.119473    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-j8tzf/perf-node-gather-daemonset-csh9j"
Sep 17 21:57:55.678947 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:55.678920    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7dbd69c77d-67kg9_48cebd9b-ef8d-46d5-8973-5d5c7a7e83fb/manager/0.log"
Sep 17 21:57:55.730200 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:55.730167    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-lnggf_bac70aa7-3806-4c7d-9dca-248f98f29612/openshift-lws-operator/0.log"
Sep 17 21:57:57.249821 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:57.249774    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-92k25_bc057dcf-d949-4624-b5c7-631b60bbc740/istio-proxy/0.log"
Sep 17 21:57:57.271750 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:57.271705    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-92k25_bc057dcf-d949-4624-b5c7-631b60bbc740/istio-proxy/1.log"
Sep 17 21:57:57.292490 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:57.292468    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8b75c8b68-8wx4k_7eb327ea-1cee-4481-8533-423762e782e0/router/0.log"
Sep 17 21:57:57.964316 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:57.964285    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-b9xfq_15c055e0-7da5-4863-90d9-c736126957ff/s3-init/0.log"
Sep 17 21:57:57.980474 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:57.980454    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-b9xfq_15c055e0-7da5-4863-90d9-c736126957ff/download-hf-model/0.log"
Sep 17 21:57:58.604583 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:58.604553    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-92k25_bc057dcf-d949-4624-b5c7-631b60bbc740/istio-proxy/0.log"
Sep 17 21:57:58.627220 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:58.627194    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-92k25_bc057dcf-d949-4624-b5c7-631b60bbc740/istio-proxy/1.log"
Sep 17 21:57:58.644020 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:57:58.643998    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8b75c8b68-8wx4k_7eb327ea-1cee-4481-8533-423762e782e0/router/0.log"
Sep 17 21:58:03.606350 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:58:03.606313    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-9mj28_08f37200-5c00-4946-8f1f-4b640e804e32/kube-storage-version-migrator-operator/1.log"
Sep 17 21:58:03.608200 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:58:03.608170    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-9mj28_08f37200-5c00-4946-8f1f-4b640e804e32/kube-storage-version-migrator-operator/0.log"
Sep 17 21:58:04.906611 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:58:04.906581    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zcsbp_5ddbfff5-cd86-477e-9152-7b0969bb0054/kube-multus-additional-cni-plugins/0.log"
Sep 17 21:58:04.925349 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:58:04.925323    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zcsbp_5ddbfff5-cd86-477e-9152-7b0969bb0054/egress-router-binary-copy/0.log"
Sep 17 21:58:04.944116 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:58:04.944092    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zcsbp_5ddbfff5-cd86-477e-9152-7b0969bb0054/cni-plugins/0.log"
Sep 17 21:58:04.963748 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:58:04.963718    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zcsbp_5ddbfff5-cd86-477e-9152-7b0969bb0054/bond-cni-plugin/0.log"
Sep 17 21:58:04.982236 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:58:04.982200    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zcsbp_5ddbfff5-cd86-477e-9152-7b0969bb0054/routeoverride-cni/0.log"
Sep 17 21:58:05.000584 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:58:05.000564    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zcsbp_5ddbfff5-cd86-477e-9152-7b0969bb0054/whereabouts-cni-bincopy/0.log"
Sep 17 21:58:05.020573 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:58:05.020547    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zcsbp_5ddbfff5-cd86-477e-9152-7b0969bb0054/whereabouts-cni/0.log"
Sep 17 21:58:05.163674 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:58:05.163597    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-r58z7_a83fb9fa-5c05-488c-bfc6-5c8e15bd62cd/kube-multus/0.log"
Sep 17 21:58:05.273210 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:58:05.273175    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-8b6dt_d42a790e-18b7-4478-b0c7-3c7cc4372fff/network-metrics-daemon/0.log"
Sep 17 21:58:05.287944 ip-10-0-139-36 kubenswrapper[2575]: I0917 21:58:05.287914    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-8b6dt_d42a790e-18b7-4478-b0c7-3c7cc4372fff/kube-rbac-proxy/0.log"